EMULAB Forum

clrmamepro [English] => clrmame Discussion => Topic started by: haynor666 on 21 June 2013, 19:53

Title: Strange 7z LZMA2 warnings
Post by: haynor666 on 21 June 2013, 19:53
clrmamepro some files in mame set packed with 7z 9.22 using LZMA2 mode give warnings. But files are fine for WinRAR or 7zip itself, even total commander 7z plugin sees them as correct. I'm using following options:

a -y -r -ms=off -mx9 -m0=LZMA2 -mfb=256 -md=256m %1 %2

clrmamepro can't move/delete those files or even decompress.

File are:
bingor2.7z problematic file inside archive is bingo roll vip2 v26.02.02_l.bin
kodja.7z problematic file inside archive is kd_20.9c
mladyhtr.7z problematic file inside archive is mlht_20.bin
sc5dndbbf.7z problematic file inside archive is p3209p11.lo
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 22 June 2013, 19:31
well, usually the errors come from the 7z sdk itself....which is..hmm..let's say...problematic here or there...not to talk about the current alpha beta whatever status of 7z....

you can try to send me the archives and I will have a closer look.
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 23 June 2013, 20:48
Ok, I've put link for these files in PM.
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 23 June 2013, 21:32
Actually, the error comes from cmpro's full archive test (when you enabled this). It compares the found crc32 from the archive header with the recalculated one by decompressing the data into memory. I will check your sent files later tomorrow (if I find a little time)...
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 24 June 2013, 19:04
Well tough luck, the sdk call SzArEx_Extract simply fails for decompressing (and so it fails for recalculating sha1/md5/crc32 and comparing them with the header).

...in other words...you have to wait till the 7z sdk gets updated....if this ever happens
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 24 June 2013, 20:35
You are using 9.22 SDK or 9.20 SDK ?
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 25 June 2013, 06:52
haynor666, sdk 9.22 is used... I debugged it a bit inside of the SzArEx_Extract function but it fails within decoding this LZMA2:17 (whatever type is this...since LZMA2 is no problem) data stream....maybe I will have a deeper look where inside the 7z code it fails...but my time is currently very limited.....

to be more concrete...it fails in 7zDec.c where the status is not LZMA_STATUS_FINISHED_WITH_MARK...."status" at that point is LZMA_STATUS_NEEDS_MORE_INPUT

        if (state.decoder.dicBufSize != outSize || lookahead != 0 ||
            (status != LZMA_STATUS_FINISHED_WITH_MARK))
          res = SZ_ERROR_DATA;

hmm....guess I need more time to debug.....
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 25 June 2013, 19:26
SDK will be probably created when next beta will show up. Just nobody knows when it be.
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 25 June 2013, 19:41
well...generally the C part of the sdk is limited (esp. when it comes to solid archives). CPP part however uses some annoying COM interface stuff.....if I got lots of time (doubful in the near future) I may look into this again....
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 04 July 2013, 21:30
some good news....I quickly looked at the CPP 7z sdk part and its way to extract/test/compress files....and hacked in a quick decompress using that...your files are decompressed correctly...

I may have some time end of this month...so most likely I switch to CPP based routines for extraction (maybe later for compression, too) of 7z (currently it uses the C part of the SDK)....blockwise decompression (less memory usage) might be a nice sideeffect....
Title: Re: Strange 7z LZMA2 warnings
Post by: oxyandy on 05 July 2013, 03:40
Quote
blockwise decompression (less memory usage) might be a nice sideeffect....

That will make 32 bit users happy !



<---- ;)
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 05 July 2013, 06:59
eh?
Title: Re: Strange 7z LZMA2 warnings
Post by: oxyandy on 05 July 2013, 07:06
You forget already ?
http://www.emulab.it/forum/index.php?topic=3474.msg13387#msg13387 (http://www.emulab.it/forum/index.php?topic=3474.msg13387#msg13387)

Bye
Great work Roman !!
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 05 July 2013, 07:46
Ah...ok...right...totally forgot that there are users which don't have at least 16GB RAM :) Actually I forgot that there are people out there which use PCs....I rarely touch it these days...

Well, I can't promise that it will be better memorywise but at least I started to have a look at the CPP SDK code....and I managed to do a quick direct extraction to disk of the problematic 7z in question....
Cleanup and adding callbacks for blockwise checksum calculation etc is next.....but again...I doubt I have time before end of this month...
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 02 August 2013, 19:51
Thanks for fix this problem  :)

I understand this is not implemented in most recent version of clrmame, right?

BTW. I hope some day solid compression will be fully supported in clrmame. Right repacking torrent7z files are big pain.
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 02 August 2013, 21:19
not yet in since I had less time as expected
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 19 December 2013, 18:49
Problem is not fixed in recent version.
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 19 December 2013, 19:07
No it's not in...since I actually did not look at it anymore....
7z is very very low priority within the very very low amount of time I got....

get more RAM instead :-)
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 19 December 2013, 23:10
I have 6 Gb :)
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 20 December 2013, 07:37
I've got 16 ;-)


I think there was a still a problem with some callback functions (e.g. to calculate sha1 on a buffer etc...)....but as I said...I haven't looked at it anymore...maybe in 2014..
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 04 April 2014, 21:12
Most recent update fixes those lzma warnings. Thanks Roman :)
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 05 April 2014, 07:45
you are welcome....see sometimes low prio things can come more quickl as expected
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 05 April 2014, 12:52
Since so much has been changed in recent versions time to test rebuild, scanner etc. :)
Title: Re: Strange 7z LZMA2 warnings
Post by: GeoMan on 08 April 2014, 22:30
I have a perfect 152 set (7z-merged) checked with a previous ClrMame using the external 7zip.exe but when i replaced the mame64.exe with the 153 version and run ClrMame 4.014a i noticed delays and many errors of the type "romxxx.7z can't be read".

Actually the roms reported by ClrMame can't be opened now. As i store the roms on a Raid 1, i checked the raid and everything reports fine.

Is it possible a bug in the 7z implementation in latest ClrMame to cause rom corruption?
Title: Re: Strange 7z LZMA2 warnings
Post by: oxyandy on 09 April 2014, 05:00
Quote
latest ClrMame to cause rom corruption
Nope no way, there would hundreds of other reports, not just you.
I assume you were using CMP to "Scan & Fix" this 152 set to 153 ?
(That's when you noticed the errors)

In "Compressor" settings CMP has a "Test archive(decompress to memory)(Scanner Only)" check box.
If you didn't actually ever run this test (not that I am saying people should ever use this all the time)
for the whole set, it may well be there were some corrupt archives already.
Is this a set you actually made completely using a previous CMP version.
Or just a downloaded 'as is' set ?

How many corrupt archives are we talking about.?
Plus there is corrupt (archive can not even display contents)
and corrupt (archive can display contents, but reports error when you try extract/modify contents)
Which type of corruption are you seeing ?
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 09 April 2014, 05:15
as oxyandy points out....can you repeat this with lets say one file, if yes send me the file plus your cmpro config (cmpro.ini and .cmp file of your used profile)...
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 09 April 2014, 07:06
and on a sidenote, yes of course things have changed....now the 7z dll is required and it does some calls via com system (create objects etc)....and if such a call fails it usually returns a "cannot open" error. I don't remember seeing or adding a "can't be read" error message though, so I expect it comes from 7z (dll/exe) itself....

would be interesting to see such a file.....
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 10 April 2014, 09:42
I've made rebuild two times using MAME 153 as a base dat. Both pass fine with word 256 and dictionary 256m using LZMA2. One pass took about 6,5 h on i5 4670k

Also fast scans (just under 5 mins) and full scans with chd, 7zip integrity, CRC32, MD5 (about 45 mins) are fine.
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 10 April 2014, 09:45
yeah....I guess it's not a general 7z failure....sounds a bit more 3rd party 7z ( 8) no I don't say the name here) may have a problem.....unless I got a file handy I can't say...or do tests...
Title: Re: Strange 7z LZMA2 warnings
Post by: GeoMan on 10 April 2014, 14:56
I had setup 152 with a previous version of ClrMame configured to use Torrent7Z (t7z.exe). When i updated to latest ClrMame i forgot to change it to the included 7z.exe.

Maybe that caused problems with 153? I changed to the included 7z.exe, fixed the damaged roms and i am now checking them with full integrity check. So far no problems (it runs for 13 hours on my i7 [email protected] )...
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 10 April 2014, 17:36
so I was right with " 8) no I don't say the name here)"
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 10 April 2014, 17:40
t7z is ok but only for recompressing for "specific" task. Using it as main compress does not make sense. Better compress with zip then recompress with t7z.
Title: Re: Strange 7z LZMA2 warnings
Post by: oxyandy on 11 April 2014, 03:14
GeoMan you may sense something going here.

The times you claim seem overly extended.
As I pointed out earlier.
Quote
"Test archive(decompress to memory)(Scanner Only)"
I know exactly what this does & how it works.
Think I have used it maybe 'once' across a whole MAME set, because I had a reason to.

Now the taboo word 't7z.exe'
There is nothing wrong with keeping sets in this format, but it really does require the correct 7z command line version & the correct strings.
(Never set t7z.exe as the program in CMP)

Doing a complete Rebuild from ZIP to 7z, regardless of cmd strings is dumb.
Regardless of the fact that some well know file sharing sites have advertised this way is the BEST.
They are wrong, well for MAME anyway, I aint going to write about all I understand here.

Keeping a set 'maintained' once in t7z with CMP is easy.
But again requires the correct 'technique' searching my posts around here, will find you 'some info'

Question time, your cmd line version of 7z & strings ?
Title: Re: Strange 7z LZMA2 warnings
Post by: haynor666 on 11 April 2014, 06:53
For better space savings 7z is the best but you get slower rom loading in mame for example c1k or neogeo  bigger roms. Also rebuild,fix, scan processes are longer. I have 3 TB disk so I have plenty of free space but this set is also used in my cab where I have only 250 gb disk so I prefer to keep set in 7z.

Please note that clrmamepro does not build solid archives. After t7z archives even smaller.

From my experience build clean set using zip then using t7z for "special" purpurose is faster than rebuild 7z and then again t7z.
Title: Re: Strange 7z LZMA2 warnings
Post by: oxyandy on 11 April 2014, 08:12
build clean set using zip then using t7z for "special" purpose is faster than rebuild 7z and then again t7z.
Yes, of course this is what I was pointing out..
Here's a bit of why.
When CMP Rebuilder finds a source ZIP archive has 10 internal files, even if all those 10 files are a match to the dat.
When Rebuilding to 7z - CMP will do this, 'file by file' 10 times for each file found matching the DAT.
1. Make a 7z with 1st found in the ZIP = result a 7z archive with 1 file.
Done
2. Make new 7z with 2nd file found in the ZIP
To do this CMP adds an extra step, extract all the contents from the first 7z archive it made,
takes that 2nd 'found' file in source ZIP, then make 'new' 7z archive with 2 files.
Done
3. Make new 7z with 3rd file found in the ZIP
CMP extracts contents of 2nd 7z archive made, takes the 3rd 'found' file in source ZIP = make 'new' 7z archive with 3 files.
Done.
And well, that's the pattern, 10 freaking times.

Rebuilder - Source archive ZIP - Destination ZIP
Extract 1st from ZIP, make 'new' ZIP with 1 file.
Extract 2nd from ZIP, add to existing ZIP now 2 files, and so on.
CMP can simply ADD to the destination ZIP file, look at all the extra work that gets cut out.

One day, if lucky enough Roman may just be kind enough to add a read ahead optimisation to CMP.
When Rebuilding - CMP will do this, 'per set' once.
Source file ZIP, 11 internal files - Destination 7z 10 files.
Read source ZIP.
File 1,2,3,4,5,6,7,8,9,10 match dat & 11 no match, end of archive ?
OK, time to build archive = results logged.
Now CMP will build a 7z once with all 10 files.
Remove Source Ticked ? OK, delete 10 from source ZIP = remainder 1

Right enough Roman ? ;)


EDIT: Oh and as you probably know extracting a 'single file' from a solid 7z is a load of extra work too !
So a bit of 'read ahead' would help when source is 7z too


Title: Re: Strange 7z LZMA2 warnings
Post by: oxyandy on 11 April 2014, 08:27
Actually I want to edit but think new post is more relevant: I want to make a correction.

Quote
When Rebuilding - CMP will do this, 'per set' once.
Now this is a bit tricky in true definition of 'per set'
To do that CMP would have to read & log entire contents of source folder before it built a single archive.
(not impossible, but definitely a bit more tricky)
So a better statement would be instead of 'per set' would be 'per source archive'
That would still give results way better than we have now.
And yeah Roman has a 'life' & we have a suitable work around already.
But we can dream right  haha
Title: Re: Strange 7z LZMA2 warnings
Post by: Roman on 11 April 2014, 08:35
Yeah....I got a life, wife, 2 sons and a real life job....which take all/most of my time...actually these days I'm thinking about switching to Windows 8.1 (yikes...do I really have to buy that thing....I wanted to skip it after Windows 7), Visual Studio 2013 Express and start all over with a C++/stl/boost lib based, no features, mame only version....keeping the DOM data in memory using xpath to access them....and doing rebuilder/scanner completely different....way more faster (due to not supporting each and any special thing or old legacy stuff)

...but maybe this is just Friday morning dreaming.....