EMULAB Forum
clrmamepro [English] => clrmame Discussion => Topic started by: ntt3 on 04 March 2020, 22:03
-
Hi all
new to the forum, even though I've been lurking around here for ages :)
I'd like to reduce the ZIP compression level apparently hard-wired into cmp...
A search netted me that the ZIP compressor tab has been removed like... 9yrs ago, and apparently there's no INI file option to tame it down - differently from RAR and 7Z.
I can see that every ZIP file produced when rebuilding (I use recompress) defaults to DeflateMaximum compression, which I assume to be a setting of 9 in the 7z DLL scale.
Long story short, the maxed out ZIP compression level takes way too much time for my frequent update batches..
Is there anything I can do for bringing it down a couple notches? Even a cmpro.ini config line (similar to the RAR and 7z entries there) would save the day for me!
Thanks in advance, and my compliments for CMP!!!
PS:
RATIONALE: I had some tests with the latest stable 7z/ZIP (deflate) compression levels:
Normal (5): Time=1, Size=100%
Maximum (7): Time=2-3x, Size=around 95-97%
Ultra (9): Time=10x (!), Size=around 90-93%
Default compression level in CMP would seem to be set to Ultra (9).. Sigh :)
-
You're right, there is no settting (not even a hidden one) which allows setting of the compression level. The old one was removed completely.
In my opinion, there is also no need to readd it.
Why? Well, let's look at the common use: "Fixing a set"
This either means you rename a set or a file in a set. Both are done without any recompression at all. You remove a file in a set. Since zip is not solid, compression does not take place either. So you come down on adding a file to a set. Again, you can use the no-recompression method and even if not, with the rather small file sizes within MAME (or similar) and modern CPUs, you have a pretty small amout of time to spend on it.
Now you say: "Rebuilder"...yes, sure, if you completely rebuild sets with recompression on, then you spend more time with it, but usually people use the rebuilder where there is no need. I read dozens of tutorials where people tend to use the rebuilder to recreate their collection over and over again when a new MAME version is out.....weird....Originally it was created to transform sets between various emulators e.g. RAINE to MAME sets or vice versa....
And actually, a lot of users like their sets to be compressed as best as possible or beyond (;-)). People have weird 7z solid mega ultra 99 path compression settings just to get 2 bytes more out of it....again....weird. Or they use post processing (actually I like to run my ZipMax program, too ;-)). Or they keep the sets decompressed and have a commandline based script which compress them after a scan....
So really....I don't see the time scaling factor of 9 versus 5 once a month for a handful of files as critical as you.....you can still switch to no compression and a post processing ;-) ....having the level in the config...shouldn't be too hard to add either....;-)
-
Thanks for your reply Roman, and for all the time you spend on CMP and supporting it!
Actually I run multiple batches every month, for a number of reasons (last but not least, formats - Retropie, anyone?) and each of them takes on my not-so-slow Win PC something in the neighbourhood of 9 hours!
Storage is cheap these days, but time is at a premium for me... if I could trade some space for some processing time, I couldn't be happier.
I sure understand it's weird asking to re-add something which was deemed useless almost a decade ago, but if you can do something... please please do it! As said, even a "minimalist" INI file option would spare me some headaches.
Otherwise I will definitely explore the "no compression, then batch-compress" option you have suggested :)
Thanks a lot!
-
I put it on the list ;-)
-
Thank you sir, much much appreciated! 8)
-
lucky you....
https://mamedev.emulab.it/clrmamepro/binaries/cmpro20200305.rar
cmpro.ini
Packer_Zip_CompressionLevel = 9
-
WOW, just WOW!!! Thank you! 8) 8) 8)
Will give it a spin this evening, will let you know how well it works in my scenario.
Thanks again
-
UPDATE
The latest rebuild batch, which originally took 9.5h, took now <4.5h (by just reducing the ZIP compression level 9->5)
Size wise, making the processed files total size = 100%, it's now 100.67%
To summarize: processing time almost cut in half, size increased by a negligible +0,67%
That's what I'd call a sterling success!
I can only recommend this trick to anyone wishing to reduce their processing time dramatically.
Thanks again Roman, you've brought a smile to my face :)
-
you mean the 3 users out there who run rebuild jobs which take longer than half an hour ;-)
hehehehe......good to hear it helped you.
-
Honestly I'm a bit puzzled as to how I could do it differently... OK it's off-topic here, but would you pls give me any pointers?
General "wisdom" on the 'net says something like: take your repository v1, add the new stuff v2, hit REBUILD and (many hours later) you get everything updated to full v2.
Especially when you need to generate different formats (split to non-merged, for instance), I don't know any ways around it...
Thanks!
-
The usual method of updating is:
Load the updated dat/profile
Scanner, run a full scan with all fix options on
After that you should only have missing stuff left
Then drag'n drop the "new" stuff into the scanner window...
(this internally runs a rebuild operation, which adds the new stuff to your collection, if your rebuilder destination points to your rompath)
So normally, on a modern system (SSD, fast CPU), a MAME update is done within minutes.....
-
Ouch. Live an learn, they say...
Never saw this flow described anywhere, I should probably get out from under that rock :D
Will try it ASAP
Many many thanks
-
it worked, as you said. Updated everything within minutes, all good :o
I'm still scratching my head about against what the initial - empty, so to speak - scan runs.
That is: yes I'm left with the difference, but the difference between the datfile at hand and.. what?
There are some other unknowns from my point of view, like where the missing stuff is being pulled from, but I can live with that EDIT: I've found your Hints and Tricks thread... got it.
All I know is that to use CMP this way I'll have to (learn how to, and) properly set all the sys variables/paths - which so far I could safely ignore, to a point.
I've been using CMP my (self-thaught) way for years, and now I understand I've been using only a fraction of its features. Never too late to learn, huh?
Thanks
PS: pls feel free to move this discussion to a new, more on-topic discussion - where it may benefit some other poor souls :)
-
"I'm still scratching my head about against what the initial - empty, so to speak - scan runs.
That is: yes I'm left with the difference, but the difference between the datfile at hand and.. what?"
erm...I don't get that part? what do you mean with "initial" scan run?
-
Yeah sorry that didn't come out well.. I meant:
When following the upgrade method you've suggested I need to load a dat/profile and then run a full scan; by the end of that scan CMP has downloaded some missing bits and pieces. What are those bits, sort of an update to the datfile it has just processed? Until then CMP didn't have any interactions with my collection, so it can't have anything to do with that.. (I've ran that full scan against a totally empty folder).
I'm just curious to understand what is the purpose of the sets (or partial sets} CMP downloaded. Enquiring minds and all that :D
-
"by the end of that scan CMP has downloaded some missing bits and pieces"
eeeeh...what? cmpro does not download anything by the end of a scan...
What makes you think that cmpro download anything?
-
Hmm no, you're right (obviously :) ): CMP isn't actually downloading anything, my bad.
Any subsequent runs didn't show that behavior again: empty folder as input, empty folder after the full scan pass.
That happened only once, and I mistakenly took it as normal behavior... I guess it was either fishing those bits out of some existing paths, or from something cached during previous full runs.
Sorry about the confusion. Thanks for your time
-
erm....if you scan an empty path you get an empty path afterwards...unless you got fix-missing enabled which looks into your setup addpaths, the rompaths and the backup path for something which might be the missing file and adds it.
But nothing, really not a single bit gets over the line (unless you use network drives ;-))
-
Indeed, that"s what I've found after my initial mistake: nothing downloaded, nothing at all..
-
Thanks Roman, really appreciate the new "Packer_Zip_CompressionLevel" option :)