EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: ZIP compression level  (Read 4748 times)

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
ZIP compression level
« on: 04 March 2020, 22:03 »

Hi all
new to the forum, even though I've been lurking around here for ages :)
I'd like to reduce the ZIP compression level apparently hard-wired into cmp...
A search netted me that the ZIP compressor tab has been removed like... 9yrs ago, and apparently there's no INI file option to tame it down - differently from RAR and 7Z.
I can see that every ZIP file produced when rebuilding (I use recompress) defaults to DeflateMaximum compression, which I assume to be a setting of 9 in the 7z DLL scale.
Long story short, the maxed out ZIP compression level takes way too much time for my frequent update batches..
Is there anything I can do for bringing it down a couple notches? Even a cmpro.ini config line (similar to the RAR and 7z entries there) would save the day for me!
Thanks in advance, and my compliments for CMP!!!

PS:
RATIONALE: I had some tests with the latest stable 7z/ZIP (deflate) compression levels:
Normal (5): Time=1, Size=100%
Maximum (7): Time=2-3x, Size=around 95-97%
Ultra (9): Time=10x (!), Size=around 90-93%
Default compression level in CMP would seem to be set to Ultra (9).. Sigh :)
Logged


Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.122 Chrome 80.0.3987.122
    • View Profile
Re: ZIP compression level
« Reply #1 on: 04 March 2020, 23:45 »

You're right, there is no settting (not even a hidden one) which allows setting of the compression level. The old one was removed completely.

In my opinion, there is also no need to readd it.

Why? Well, let's look at the common use: "Fixing a set"
This either means you rename a set or a file in a set. Both are done without any recompression at all. You remove a file in a set. Since zip is not solid, compression does not take place either. So you come down on adding a file to a set. Again, you can use the no-recompression method and even if not, with the rather small file sizes within MAME (or similar) and modern CPUs, you have a pretty small amout of time to spend on it.

Now you say: "Rebuilder"...yes, sure, if you completely rebuild sets with recompression on, then you spend more time with it, but usually people use the rebuilder where there is no need. I read dozens of tutorials where people tend to use the rebuilder to recreate their collection over and over again when a new MAME version is out.....weird....Originally it was created to transform sets between various emulators e.g. RAINE to MAME sets or vice versa....

And actually, a lot of users like their sets to be compressed as best as possible or beyond (;-)). People have weird 7z solid mega ultra 99 path compression settings just to get 2 bytes more out of it....again....weird. Or they use post processing (actually I like to run my ZipMax program, too ;-)). Or they keep the sets decompressed and have a commandline based script which compress them after a scan....

So really....I don't see the time scaling factor of 9 versus 5 once a month for a handful of files as critical as you.....you can still switch to no compression and a post processing ;-) ....having the level in the config...shouldn't be too hard to add either....;-)


« Last Edit: 04 March 2020, 23:47 by Roman »
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #2 on: 05 March 2020, 10:02 »

Thanks for your reply Roman, and for all the time you spend on CMP and supporting it!
Actually I run multiple batches every month, for a number of reasons (last but not least, formats - Retropie, anyone?) and each of them takes on my not-so-slow Win PC something in the neighbourhood of 9 hours!
Storage is cheap these days, but time is at a premium for me... if I could trade some space for some processing time, I couldn't be happier.
I sure understand it's weird asking to re-add something which was deemed useless almost a decade ago, but if you can do something... please please do it! As said, even a "minimalist" INI file option would spare me some headaches.
Otherwise I will definitely explore the "no compression, then batch-compress" option you have suggested :)
Thanks a lot!
« Last Edit: 05 March 2020, 10:21 by ntt3 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.122 Chrome 80.0.3987.122
    • View Profile
Re: ZIP compression level
« Reply #3 on: 05 March 2020, 11:47 »

I put it on the list ;-)
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #4 on: 05 March 2020, 11:54 »

Thank you sir, much much appreciated!  8)
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.122 Chrome 80.0.3987.122
    • View Profile
Re: ZIP compression level
« Reply #5 on: 05 March 2020, 14:43 »

lucky you....

https://mamedev.emulab.it/clrmamepro/binaries/cmpro20200305.rar

cmpro.ini
Packer_Zip_CompressionLevel = 9
« Last Edit: 05 March 2020, 14:43 by Roman »
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #6 on: 05 March 2020, 15:15 »

WOW, just WOW!!!  Thank you! 8) 8) 8)
Will give it a spin this evening, will let you know how well it works in my scenario.
Thanks again


Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #7 on: 06 March 2020, 13:49 »

UPDATE
The latest rebuild batch, which originally took 9.5h, took now <4.5h (by just  reducing the ZIP compression level 9->5)
Size wise, making the processed files total size = 100%, it's now 100.67%
To summarize: processing time almost cut in half, size increased by a negligible +0,67%
That's what I'd call a sterling success!
I can only recommend this trick to anyone wishing to reduce their processing time dramatically.
Thanks again Roman, you've brought a smile to my face :)
« Last Edit: 06 March 2020, 13:49 by ntt3 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.132 Chrome 80.0.3987.132
    • View Profile
Re: ZIP compression level
« Reply #8 on: 06 March 2020, 14:14 »

you mean the 3 users out there who run rebuild jobs which take longer than half an hour ;-)
hehehehe......good to hear it helped you.
« Last Edit: 06 March 2020, 14:14 by Roman »
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #9 on: 06 March 2020, 16:09 »

Honestly I'm a bit puzzled as to how I could do it differently... OK it's off-topic here, but would you pls give me any pointers?
General "wisdom" on the 'net says something like: take your repository v1, add the new stuff v2, hit REBUILD and (many hours later) you get everything updated to full v2.
Especially when you need to generate different formats (split to non-merged, for instance), I don't know any ways around it...
Thanks!
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.132 Chrome 80.0.3987.132
    • View Profile
Re: ZIP compression level
« Reply #10 on: 06 March 2020, 17:25 »

The usual method of updating is:
Load the updated dat/profile
Scanner, run a full scan with all fix options on
After that you should only have missing stuff left
Then drag'n drop the "new" stuff into the scanner window...
(this internally runs a rebuild operation, which adds the new stuff to your collection, if your rebuilder destination points to your rompath)

So normally, on a modern system (SSD, fast CPU), a MAME update is done within minutes.....
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #11 on: 06 March 2020, 20:06 »

Ouch. Live an learn, they say...
Never saw this flow described anywhere, I should probably get out from under that rock :D
Will try it ASAP
Many many thanks
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #12 on: 06 March 2020, 21:48 »

it worked, as you said. Updated everything within minutes, all good  :o
I'm still scratching my head about against what the initial - empty, so to speak - scan runs.
That is: yes I'm left with the difference, but the difference between the datfile at hand and.. what?
There are some other unknowns from my point of view, like where the missing stuff is being pulled from, but I can live with that  EDIT: I've found your Hints and Tricks thread... got it.
All I know is that to use CMP this way I'll have to (learn how to, and) properly set all the sys variables/paths - which so far I could safely ignore, to a point.
I've been using CMP my (self-thaught) way for years, and now I understand I've been using only a fraction of its features. Never too late to learn, huh?
Thanks

PS: pls feel free to move this discussion to a new, more on-topic discussion - where it may benefit some other poor souls :)
« Last Edit: 06 March 2020, 23:39 by ntt3 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.132 Chrome 80.0.3987.132
    • View Profile
Re: ZIP compression level
« Reply #13 on: 07 March 2020, 11:43 »

"I'm still scratching my head about against what the initial - empty, so to speak - scan runs.
That is: yes I'm left with the difference, but the difference between the datfile at hand and.. what?"

erm...I don't get that part? what do you mean with "initial" scan run?
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Linux Linux
  • Browser:
  • Chrome 80.0.3987.132 Chrome 80.0.3987.132
    • View Profile
Re: ZIP compression level
« Reply #14 on: 07 March 2020, 13:40 »

Yeah sorry that didn't come out well.. I meant:
When following the upgrade method you've suggested  I need to load a dat/profile and then run a full scan; by the end of that scan CMP has downloaded some missing bits and pieces. What are those bits, sort of an update to the datfile it has just processed? Until then CMP didn't have any interactions with my collection, so it can't have anything to do with that.. (I've ran that full scan against a totally empty folder).
I'm just curious to understand what is the purpose of the sets (or partial sets} CMP downloaded. Enquiring minds and all that :D
« Last Edit: 07 March 2020, 13:46 by ntt3 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.132 Chrome 80.0.3987.132
    • View Profile
Re: ZIP compression level
« Reply #15 on: 07 March 2020, 16:36 »

"by the end of that scan CMP has downloaded some missing bits and pieces"

eeeeh...what? cmpro does not download anything by the end of a scan...
What makes you think that cmpro download anything?
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 73.0 Firefox 73.0
    • View Profile
Re: ZIP compression level
« Reply #16 on: 07 March 2020, 18:30 »

Hmm no, you're right (obviously :) ): CMP isn't actually downloading anything, my bad.
Any subsequent runs didn't show that behavior again: empty folder as input, empty folder after the full scan pass.
That happened only once, and I mistakenly took it as normal behavior... I guess it was either fishing those bits out of some existing paths, or from something cached during previous full runs.
Sorry about the confusion. Thanks for your time
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 80.0.3987.132 Chrome 80.0.3987.132
    • View Profile
Re: ZIP compression level
« Reply #17 on: 07 March 2020, 18:56 »

erm....if you scan an empty path you get an empty path afterwards...unless you got fix-missing enabled which looks into your setup addpaths, the rompaths and the backup path for something which might be the missing file and adds it.
But nothing, really not a single bit gets over the line (unless you use network drives ;-))
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Linux Linux
  • Browser:
  • Chrome 80.0.3987.132 Chrome 80.0.3987.132
    • View Profile
Re: ZIP compression level
« Reply #18 on: 07 March 2020, 19:46 »

Indeed, that"s what I've found after my initial mistake: nothing downloaded, nothing at all..
Logged

vicmarto

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 21
  • Operating System:
  • Mac OS X 10.15.3 Mac OS X 10.15.3
  • Browser:
  • Safari 13.0.5 Safari 13.0.5
    • View Profile
Re: ZIP compression level
« Reply #19 on: 23 March 2020, 15:24 »

Thanks Roman, really appreciate the new "Packer_Zip_CompressionLevel" option   :)
Logged
Pages: [1]   Go Up
 

Page created in 0.116 seconds with 20 queries.