EMULAB Forum

clrmamepro [English] => clrmame Discussion => Topic started by: Roman on 28 February 2014, 18:42

Title: 7z blockwise memory decompress
Post by: Roman on 28 February 2014, 18:42
As you might notice from the amount of releases I'm pretty busy with real life.....and so looking at 7z is nearly impossible at the moment...

Looking at 7z currently means: trying to get the c++ part of the 7z SDK to work...the old routines use the C core of the SDK, which is limited when it comes to blockwise operations (everything is decompressed to memory/file in one go) and > 4GB support.

While I had no problem to use the general reading/extracting mechanisms of the C++ COM part I'm still looking at way to have a callback ready which is able to update calculated hash values or -most likely the same callback- decompress data blockwise to memory (for testing purposes)...

I might have overseen such a callback due to my lack of time.....but I only found some update callback which can e.g. be used to print out a progress percentage value...but not giving me access to a memory block/size (to update my hash calcs etc)...

So....if anyone knows where to find this...you're more than welcome...
Title: Re: 7z blockwise memory decompress
Post by: Roman on 03 March 2014, 13:03
ok...got the answer from the 7z author himself...


"
Client7z.cpp:
CArchiveExtractCallback::GetStream
_outFileStreamSpec = new COutFileStream;
CMyComPtr<ISequentialOutStream> outStreamLoc(_outFileStreamSpec);
if (!_outFileStreamSpec->Open(fullProcessedPath, CREATE_ALWAYS))
{
  PrintError("Can not open output file", fullProcessedPath);
  return E_ABORT;
}
You must create object of your class, that implements ISequentialOutStream and calculates Hash
"



now I only need to time to do so :-)
Title: Re: 7z blockwise memory decompress
Post by: Roman on 05 March 2014, 08:53
hmm...ok...seems that after years I start to understand that CPP core.... :-)
Title: Re: 7z blockwise memory decompress
Post by: Roman on 07 March 2014, 10:39
ok...7z fans out there..

I got the blockwise decompress to memory with hash calculation hooked up...nice...works as it should...next step is to change the actual decompress to hd to the CPP SDK core, too...ain't hard..it's nearly the same code...

Currently I don't know how much time I got before Easter...but looks like you can be sure that the next version won't have any memory issues with (large) 7z files anymore...this should also increase speed for 7z operations which use decompress/hashcalcs.

Compression will still be done via external packer though...(now that I slightly understand how this stuff works I might look at that some day, too).

Well...keep you updated....



Little update:

File extract to HD works, too.... so now on the to do list is actually only testing (esp. with corrupt 7z files)...and maybe I change the table of contents reader also to the new cpp 7z sdk core.....
As mentioned before...I nearly don't have any free time...so don't expect anything anytime soon...maybe a preview which you can play around with...we'll see.....Currently I'm happy that I finally got this COM CPP core working....yieeeha....and now...a weekend...without any coding...
Title: Re: 7z blockwise memory decompress
Post by: Roman on 10 March 2014, 22:09
another entry in this 7z diary....completely replaced the 7z C SDK usage with the 7z C++ COM part....i.e. central dir reader, extract to disc and extract to memory with hash calculation is now based on the current C++ SDK core....which is...erm...cool... ;-)
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 10 March 2014, 23:27
Quote
which is...erm...cool... ;-)

Yes, very cool :D
If you need a tester....

I have some archives put aside that have been previously troublesome.
Title: Re: 7z blockwise memory decompress
Post by: Roman on 11 March 2014, 05:30
yeah yeah.  test version is on its way.....
Title: Re: 7z blockwise memory decompress
Post by: oddi on 11 March 2014, 10:11
Tester #2 waiting...:)
Title: Re: 7z blockwise memory decompress
Post by: Roman on 11 March 2014, 10:33
thanks for the interest....
so...test scenarios could be something like...

> 4gb 7z files
corrupt 7z files
7zfiles with only folder entries
7z without specified file attributes, sizes or crc32 (yes, such things do exist..I'm still looking for the commandline flags to create them...)
utf8 stored filenames and if they are read in correctly...

I expect to put out a test version this week...keep in mind, this does not change slow operations on solid archives...so for now this should resolve issues with huge 7z files where the old C based code wanted to do everything in memory (i.e. it decompressed 4GB to memory to calculate the hash at once....).....and of course the internal use to get rid of the old C based part...
Title: Re: 7z blockwise memory decompress
Post by: Roman on 11 March 2014, 20:34
here we go....

http://mamedev.emulab.it/clrmamepro/binaries/cmp20140311.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmp20140311.rar)
Title: Re: 7z blockwise memory decompress
Post by: Starshadow on 12 March 2014, 00:21
Ready for bugs? :) Using the 20140311 64-bit build, running a scan with the 'Test archive' option enabled, I get nothing but 'error while unpacking 7z file' in the Warnings window. The archives test fine in 7zip. Archive size or algorithm doesn't seem to matter.
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 03:52
Ok here we go,
Starshadow, I agree:
The Warning window does appear with 'Test archive (decompress to memory) (Scanner only)' ticked.

(http://ndsxdelta.no-ip.org/test_archive.png)
(http://ndsxdelta.no-ip.org/err_unpack.png)

But as far as:
Quote
I get nothing but 'error while unpacking 7z file'
CMP actually does completely extract and check each file & the test passes !
So the 'get nothing' is a little extreme.
It seems CMP is falsely reporting the error in the warning window, but completing the task successfully anyway.

(http://ndsxdelta.no-ip.org/dec_crc.png)

So warning yes, Task complete without error (on 'integrity tested = ok archive' !)

Roman, now for me I was able to completely create a DIR2DAT of:
268 files = 3.5Gb worth of internal archives in a single 14Mb Solid 7z.
Dat worked perfectly, thank you !
Previously this was not possible with CMP32 running on 32bit OS !
(Neither creating the DAT nor Scanning it)
So yeah, very cool  8)

:More to come later as I complete these tests, next check - scan a hex editor hacked 7z archive (corrupt download simulation)
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 05:19
BAD NEWS.
As above,  'Test archive (decompress to memory) (Scanner only)' ticked
I cleared cache, even deleted the dat fully and started over.

On my now hacked 7z (Corrupt archive, open-able but fails 'Test Archive' check in both 7z & WinRar)
Does not fail in CMP with 'Test archive (decompress to memory) (Scanner only)' ticked
It should be failing ! (Only 124 of 268 Original files are still good)
Scan completes 'Green'
Using 7z GUI & WinRAR GUI, only a certain percentage of the files can be extracted from the archive.

WinRar extracts 124 of 268 original files (All perfect hashes which are fully rebuild-able)
(WinRar the winner, if the file doesn't match stored hash, winrar wont extract it - unless forced in 'Keep Broken...')

7z extracts ALL 268 of 268 files, only 124 which still have correct hash.
(Long time known bug in 7z GUI (cmd too ????)- I should really report it)

Anyway back on track, dragging & dropping my corrupt 7z onto Rebuilder.
results are as expected 124 files were rebuilt, 144 (corrupt skipped)
However: No warning of corruption was shown in CMP.
Good to know CMP can still make use of the 'good' files, in my bad 7z archive.
The good files are the first 124 packed files in order, the 144 bad the remainder.
Previous CMP builds would fail with 7z: ERROR SZ_ERROR_MEM ->

Repeating 'Test archive (decompress to memory) (Scanner only)'
this time with "Decompress ROM & check SHA1/MD5" ticked in 'HASH & CHD'
Same "error while unpacking 7z file" in warning window..
No incorrect files are reported.
Actually, this time 'Files to go' = 1, and it's just stuck there - wont complete)
The corruption was NOT detected/reported ! :(

Time to wait for next build..
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 05:49
Ok so I couldn't leave it alone,

I took my 'Good 7z solid archive' and removed a single entry from the dat.
Now 1 file should be removed from my archive.
Ran Scanner, it instantly picked up the unwanted entry & asked me if I wished to remove..
I said yes, CMP did it's thing, file was removed & newly created archive tests just fine.
So that works.
EDIT: Oh yeah, while it seems I may be stating the obvious here, this was NOT possible with previous CMP builds. A huge improvement.

Hey Readers !
Some-one test the LARGE 7z archives (over 4gb) that failed hash checks previously, post results..
But remember, this is a TEST build of CMP, so any damage to your set should be avoided by working with a set/file copy not the originals, huh.
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 07:42
> 4gb 7z files (Waiting for someone else to report)
corrupt 7z files (FAIL ! = Corruption not detected)
7z files with only folder entries (Multiple tests - all passed)
No crash when file with 'folder only entries' was in Rebuilder path:
No problems when 'folder only entries' in Set folder 'Scanner'
Both cases - CMP Reported:
Corrupt Archive File:Q:\Please DAT Me.7z | Reason: NO ENTRIES
7z without specified file attributes, sizes or crc32 ( I know of CMP switch in t7z - but never seen an archive without CRC or size)  :o
utf8 stored filenames  (Multiple tests - all passed)


All normal operations like:
Renames, deletes, additions to destination files, removal from source files etc. (inc utf-8)
Illegal dates.
All seems fine here.
Title: Re: 7z blockwise memory decompress
Post by: Roman on 12 March 2014, 08:07
ok thanks...I will check the reported things tonight (now...real life job...and time for a coffee)....

highly appreciate the testing from all of you...

and yeah...still think the C++ SDK switch is cool :)
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 12:44
7z files with only folder entries (Multiple tests - all passed) Hmm maybe not, read on.
I just wanted to add, I even did a Rebuilder run, where the destination folder contained a 7z Solid Archive with only empty folders inside..
CMP Rebuilder perfectly added all the files to the Archive, so now it contains Empty Folders & files.

So I just tried "Scanner" to detect and remove those "New Folder", "New Folder (1)" Entries..
Which of course are not in the dat..
FAIL, it seems they are not even seen

Ok, next test: A quick edit to my DAT to add some sub-folder paths to this archive, see how it's handled.

The Result:
CMP Noted the folder additions (via DAT edit) "Scanner" made the correct renames to include the Folders.
Great that worked..
Manually checked the archives, the files are there in their Sub-Folders.

Ok, another DAT edit to now change the Names of those Sub-Folders in the DAT to something else..
CMP is asking to Rename those Sub-Folders to the new names made in the DAT edit, good.
But, still NOT seeing the EMPTY folders as unneeded.
Title: Re: 7z blockwise memory decompress
Post by: Starshadow on 12 March 2014, 13:07
CMP actually does completely extract and check each file & the test passes !
So the 'get nothing' is a little extreme.
It seems CMP is falsely reporting the error in the warning window, but completing the task successfully anyway.
I tested on some very large >4GB 7z files first. They failed with the aforementioned warning before enough time had passed to simply read the entire archive from disk let alone test it. When I tested smaller files, I got the exact same behavior. I didn't see any indication that any of them successfully tested.
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 13:12
Starshadow, what about without -  'Test archive (decompress to memory) (Scanner only)' ticked ?
(or in Scanner - 'Hash & CHD' - "Decompress Rom & CRC/SHA Check" checked.)
Just a regular 'New Scan' ?
Title: Re: 7z blockwise memory decompress
Post by: Roman on 12 March 2014, 13:32
Looks like it will be a long night :)

Regarding the empty folders....I wonder how old cmpro reacts....

Regarding "without crc32"...that was easy...a 0 byte file is stored without any checksum or lengh information (not even == 0).

So...does this list summarize it so far?

- test archive works but may return a wrong information
- corrupt 7z files are not always marked as being corrupt (-> message in warnings window or something)
- empty folders are not reported as unneeded
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 14:21
Regarding the empty folders....I wonder how old cmpro reacts....
Yeah, guess what old 4.012b CMP isn't seeing/removing them either.
I 'think' I have noticed that maybe since a few CMP revisions that "Rebuilder"
while removing files housed in a sub-folder in rar, that an empty folder remains after file is removed from it..
Did CMP previously remove the file & folder if removing the rebuilt file, left the folder empty ?
I guess I had better check. But something does 'feel' different there.


- test archive works but may return a wrong information
Seemed to work on OK archives, but test on corrupt files is showing them as OK too.
- corrupt 7z files are not always marked as being corrupt (-> message in warnings window or something)
Corrupt archives are always testing as OK, they should fail 'Test'.

- empty folders are not reported as unneeded
Yeah, but this behaviour is the same on 4.012b 'Official Release' - I just checked
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 12 March 2014, 14:53
 :-X
Gee, just tried 4.08b - it is not seeing the 'Not in DAT', empty folders in the 7z archive either.

Alright, converted my 7z archive to zip, cleared cache,
'New Scan' now I get prompted to Remove the not in dat, empty folders.

Summary "Scanner" with 4.08b:
1. ZIP empty, not in dat folders, are seen and removed.
2. RAR empty, not in dat folders, are seen and removed. (Inc. those double pesky folder entries WinRAR exe makes)
3. 7z  empty, not in dat folders, are NOT even seen, so of course, no prompt to remove them..

Guess I found an old bug !
haha

See something changed with WinRAR!
Summary "Scanner" with 4.012b 'Official Release':
1. ZIP empty, not in dat folders, are seen and removed.
2. RAR empty, not in dat folders, are NOT even seen, so of course, no prompt to remove them..
3. 7z  empty, not in dat folders, are NOT even seen, so of course, no prompt to remove them..

Title: Re: 7z blockwise memory decompress
Post by: Starshadow on 12 March 2014, 16:27
Starshadow, what about without -  'Test archive (decompress to memory) (Scanner only)' ticked ?
(or in Scanner - 'Hash & CHD' - "Decompress Rom & CRC/SHA Check" checked.)
Just a regular 'New Scan' ?
New Scan without either option checked works just fine. The "Decompress Rom & Check..." options appear to work fine as well.
Title: Re: 7z blockwise memory decompress
Post by: Roman on 12 March 2014, 19:15
ah ok...the detection of obsolete subfolders is limited internally to zip....since removing them only works safely in zip...for rar/7z it might remove more than just the entry for the folder... it was disabled with 4.011a

so this build:

http://mamedev.emulab.it/clrmamepro/binaries/cmp20140312.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmp20140312.rar)

should fix your other reported things...

I may have a look at the empty folder things in the next days...
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 13 March 2014, 01:18
Hi Roman,
I am still playing with the new build.
First impressions are it seems you have resolved everything reported..
Corrupt archives are shown now (both test & decompress.........), via "Scanner"
but still no msg when a corrupt 7z is found in source path (No big deal anyway) via "Rebuilder"
I could suggest a few 'tweaks' - but also feeling lazy at this point..
I will play further on & off, across the day - if anything drastic is noticed, I will post.
Otherwise, it will be a case of no news, is good news.

7z Over 4 Gigabyte Archives .. I wont be testing.. so wait for feedback from someone for those..
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 13 March 2014, 08:53
Hello from "64 Bit Land",  :(
I am seeing cmpro64 not working as it should too.
Wont Rebuild to 7z or Scan 7z...
Sorry, no time left, work & life calls

EDIT: I had almost no time at all to do much testing with either the 'new' 32 or 64 builds today..
Only a few minutes with either.

Title: Re: 7z blockwise memory decompress
Post by: Roman on 13 March 2014, 11:03
hmm...sounds weird...have no issue...will check later this evening...
Title: Re: 7z blockwise memory decompress
Post by: Starshadow on 13 March 2014, 12:00
cmp20140312 64-bit solved the issue I reported when running a scan with the 'Test Archives' option turned on, but as oxyandy alluded to, now when running a New Scan with the 'Decompress ROM & Check...' options enabled, I get 'error while unpacking 7z file:' and 'Archive Header CRC32 doesn't match CRC32 of decompressed data for file:' for every archive. Rebuilding to/from 7z seems to be ok to me though, he'll have to provide more info on what he's seeing there.
Title: Re: 7z blockwise memory decompress
Post by: Roman on 13 March 2014, 12:03
yeah yeah....somthing's spooky there....will check it tonight...thanks for the support
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 13 March 2014, 12:36
Hi Starshadow,
Again I admit both attempts I made with 'today's new builds' were both very hurried..
I use the 7z.exe from 9.28 alpha (on 32 bit) cause that version seemed the best when I made lots of tests long ago.
The x64 test a few hours ago was with 7z.exe from 9.25 x64, cause that was all I could find quickly..
What cmd version of 7z.exe are you using ?
I have promised Roman in about another 40 minutes I will confirm a more serious test with x64..

Yeah the new version CMP32 "Decompress & check..." is not working as it should, either..
It worked on the previous version.. (well sort of)
But 'test' in compressor setting now seems to work ok.. confirmed.

PS. I have since tracked down x64 7z.exe from 9.28 alpha, so I will try with that..
Title: Re: 7z blockwise memory decompress
Post by: oddi on 13 March 2014, 13:17
Hello guys, i'm happy see hard job between cmp and 7z.
Me test cmp x64 and 7z 9.32 x64 ( replace 9.28alpha ) for test.
Old bug when cmp not rename roms inside archive is fixed.
Btw - very very strange problem :
first 2 sets from mame: 005.7z and 10yard.7z :
If rename 005.7z to any name, example: 00523blabla.7z , cmp scan and ask for rename
but rename 10yard.7z to any name : example 10yardblabla.7z , cmp wanna remove file !?!
if me give permission for remove after that no problem with restore with rebuild.
Other little problem: maybe this problem only from 7z side :
if test any t7z archive with 7z 9.32 x64:
7z report: "There are data after the end of archive"
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 13 March 2014, 13:58
Hi Roman,
x64 CMP seems fine now, only thing it is doing which it shouldn't is
In "Scanner" "Hash & CHD" "Decompress ROM & check...." is producing a 7z error msg..
(Same with CMP32)
AH, now I know why I didn't see this earlier and maybe why you haven't seen it too !
I was rushing so didn't wait till (Files to go =1)

OK, I click "New Scan" on a folder with a single 7z archive, this archive has 50 internal files.
Files to go counts down from 50,
NO error message is produced.
49, 48, 47, 46 ..all the way down to... 2 NO WARNING WINDOW !
Then once "Files to go", reaches ONE,
THEN the warning window appears.
"Files to go", STAYS as ONE,
Then it one by one produces an error message for each internal file.



Hi Oddi,
Don't use GUI of 7z 9.32 x64 and you wont get "There are data after the end of archive"
These are t7z (torrent 7 zip) archives..
Use an older GUI, not sure which version, try a different 7z GUI version and see,
then no more "There are data after the end of archive" message

Oddi you say cmd line EXE of 7z v9.32 is working OK with CMP ?
Ok, I will give it a try, thanks
Title: Re: 7z blockwise memory decompress
Post by: Roman on 13 March 2014, 18:36
ah silly me...pretty obvious error...fixed now...I will now check the (unneeded) folder stuff so this might get reenabled....expect a new test version later tonight....
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 13 March 2014, 18:40
Ah good,
so you made sense of my post, haha
2.39 AM my time, being ordered to bed by the boss..
Great work, yet again !
Thx Roman.
Title: Re: 7z blockwise memory decompress
Post by: Roman on 13 March 2014, 21:38
ok here we go...this should fix the problems :)

http://mamedev.emulab.it/clrmamepro/binaries/cmp20140313.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmp20140313.rar)

...plus I've reenabled the detection of obsolete folder entries in rar/7z and made the removal of such entries via external packers more safe (in the past the external packers killed the folder...no matter what's inside :))
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 14 March 2014, 02:35
I am 'Awake'.
You think it's bug free this time ?
I'll keep testing !
I'll test everything..
From..
Source file 7z (internal files total 50) 25 match the DAT.
And Rebuild it..
will remaining source archive be not corrupt with it's remaining 25 Files  ?
Destination file ok too ?
To..
Destination files, with sub-folders 'as defined in the DAT',
then move the files around with manual Notepad++ DAT edits etc, etc
Test the 'Empty Folders' & everything else I can think of.

Right now, using CMP32, I see yesterday's bugs are indeed gone !
This 7z update is letting me do things with 7z archives that previously would fail on a 32bit OS
It really is a wonderful advancement !

IF you have 'repaired' the 'Empty Folder' removal..
Then only with 'this' release, will CMP be truly honouring the DAT in relation to DAT/Archive Sub-Folder entries.
Across any archive type, ZIP, 7z & Rar -  a First for CMP!
Honouring the DAT faithfully is, of course, what every 'ROM Manager' should do.
So I think this is SUPER important update !

Starshadow please,
Some feedback if you will, for the massive over 4Gb archives, even with the 32Bit CMP if possible.
I just don't have any datted 4Gb 7z's in any of my collection.
Title: Re: 7z blockwise memory decompress
Post by: oddi on 14 March 2014, 03:00
CMP x64 and 7zip 9.32 - not rebuild roms with drag and drop, cmp report error. Back to 9.28 - all is ok.
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 14 March 2014, 03:17
Thanks for feedback Oddi !
Right now I am making use of 7z.exe (7z.exe command line v9.32 -32 bit)
It seems to be working.. (But for sure I will keep this in my thoughts)

Roman,
I found a bug, where CMP broke the 7z archive with duplicate filenames. (Better feedback later)
As I warned, I will test everything, my goal - break CMP's working operations.
I will produce 2 DAT files & sample TEST 7z archives for you to REPEAT the operation, later..
Step by Step testing first.

CMP is great though, it is seeing the dupe entry & reporting it.

(http://ndsxdelta.no-ip.org/7z_dupe.png)

Of course now the error is there, no further processing can be made on this archive  ;D

Dumped my broken archive.
Started fresh,
Maybe I am seeing the same as you here Oddi ?  (7z.exe command line v9.32 -32 bit)

(http://ndsxdelta.no-ip.org/Re_broke.png)

I'll roll-back to v9.28, see if error goes.
The Result: Yes, Rebuilder message gone..
I will swap back to v9.32 - confirm it returns
The Result: No, same operation as before, this time it continues. Damn
Test continues on v9.32

Title: Re: 7z blockwise memory decompress
Post by: Roman on 14 March 2014, 07:59
so we have an issue with 7z.exe 9.32...and one with double file entries in a 7z file (what the heck..this should not exist anyway...)....right?
Title: Re: 7z blockwise memory decompress
Post by: Roman on 14 March 2014, 08:37
I guess I can look at it in 10 to 12 hours....
Title: Is the bug is specific to 7z only
Post by: oxyandy on 14 March 2014, 09:11
Is the bug is specific to 7z only ?
OK, let's answer that question.
First, let's explain ORIGINAL archive has 3 files.
(http://ndsxdelta.no-ip.org/orig_.png)

2 at Root level and one in a subfolder (file is same CRC as one of the root files)
Dat should remove the subfolder & it's contents.
Instead a rename is taking place, creating a dupe entry.

3 TESTS, ZIP, RAR, 7z
ZIP
No Bug, after "Scanner" is complete, archive is perfect 2 files inside.
(http://ndsxdelta.no-ip.org/zip_wins.png)

RAR
Yes, after "Scanner" is complete, archive has 3 files inside. (inc one dupe)
Another "Scanner" "New Scan" test, does NOT produce a warning message.
Can I consider no warning of dupes a bug too ? ZIP & 7z report this error, I checked.
(http://ndsxdelta.no-ip.org/rar_um.png)

7z
Yes, after "Scanner" is complete, archive has 3 files inside. (inc one dupe)
Another "Scanner" "New Scan" test, does produce a warning message.
(http://ndsxdelta.no-ip.org/7z_vs_rar.png)
(http://ndsxdelta.no-ip.org/7z_um.png)
Title: Re: 7z blockwise memory decompress
Post by: Roman on 14 March 2014, 09:52
well...it's what the external packers do if they readd a file which already exist. zip/rar removes the one instance and 7z seems to simply add it...so you got 2...

will check if there is a cmdline option to force a replace....or I need to manually remove it first...will check that later
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 14 March 2014, 10:13
I will leave it in your very capable hands.
Here are 3 perfect original archives, one of each type - zip, rar, 7z to do the tests with.
And the required DAT.
"Scanner" will break the Rar & 7z archive (see above) - zip will be fine.
Title: Re: 7z blockwise memory decompress
Post by: Starshadow on 14 March 2014, 17:17
Starshadow please,
Some feedback if you will, for the massive over 4Gb archives, even with the 32Bit CMP if possible.

>4GB 7z archives have no unique problems in cmp20140313 both 32 and 64 bit. Testing, hash checking, and extracting for rebuild all seem to be working fine for them. The hash checking could be faster for large files if it used multi-threading and multiple buffering, but that's a discussion for another day :-X. Thanks for your hard work Roman!
Title: Re: 7z blockwise memory decompress
Post by: Roman on 14 March 2014, 21:53
http://mamedev.emulab.it/clrmamepro/binaries/cmp20140314.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmp20140314.rar)


btw...I don't have any issues with 7-Zip [64] 9.32 alpha  (2013-12-01) (64bit)
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 15 March 2014, 01:02
Very cool  8)

Fix confirmed !
Now oxy rushes off to work !
Title: Re: 7z blockwise memory decompress
Post by: oddi on 15 March 2014, 08:46
Many tnx guys ! :)
Roman write here your 7z settings in cmp. Now me repalce with new cmp forum build and reinstall 7zip 9.32 x64, but not have time for test. Later test it .tnx :)
Title: Re: 7z blockwise memory decompress
Post by: Roman on 15 March 2014, 12:26
I'm using the default ones
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 15 March 2014, 14:48
oddi,
Use the same 7z cmd line settings as you used previously.
The 'Default' setting with modern 7z.exe will create LZMA2 archives, LZMA2 can not be processed by t7z..
Setting t7z.exe directly in CMP's settings is just crazy.
So post processing t7z.exe with after CMP "Scanning & Fixing" is unavoidable.
Hence the reason I set -mx0
Plus the way CMP works 'as current' by adding 'file by file' on a Rebuilder run, so using higher 7z compression is a waste of time & resources..
I would never suggest anyone try Rebuild to 7z (a whole set)
This is why I call my 7z cmd settings 'Maintenance Settings" for using CMP with an already 7z & merged set.
Someone wishing to convert a MAME split & zip set to merged 7z, should use ZIP and "Scanner" to merge & Fix FIRST.
Then send the 'ROM Root' to t7z.exe (or Drag, Drop folder onto t7z.exe) to convert to t7z..
After complete, then configure CMP to work with 7z.
(I will re-t7z the 'ROM Root' occasionally, definitely not with every dat update)
Maybe if Roman makes some core changes 'one day', well that advice might change  ;D

Roman,
Monday I will continue to test everything I can think of - starting over again with today's build.
Thanks for the effort you have put in, it's really appreciated.
I know I have worked with you fine-tuning CMP to be the best it can be, and you haven't rejected anything.
The new CMP opens up 7z usage, on a 32bit OS which couldn't be achieved previous to the latest updates.
Champion you are !

Starshadow,
+1 for your effort on checking the >4Gb 7z, both 32 & 64
Title: Re: 7z blockwise memory decompress
Post by: oddi on 15 March 2014, 17:41
Oxy tnx for the info, me use old "best" cmp 7z settings :). Now start activity use cmp x64 + 7z 9.32 x64, if found any bug report here. Tnx for all :)
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 15 March 2014, 18:12
You can use the 7z.exe (cmd line program) from 9.32 x64...
Just copy & paste the 7z.exe & put it in CMP's program folder..
(Path in settings 'default' 7z.exe)

Because, if you don't want the warning about "Information past the end of file" that t7z files cause,
then use an 'Older Version' of the 7z UI, sorry not sure which one, you will need to try yourself..

I haven't tested 7z.exe 9.32 much, it seems to work OK..
On Monday I will use it exclusively with CMP and report any issue.
I haven't used it enough to be confident there is any difference to using 9.28..
When I did the testing with 9.28, the 'latest alpha' at that time was 9.29 and that did not work as well as 9.28.
That's about all I remember, haha.
Now latest alpha is 9.32..
I am not sure there is any reason to update from 9.28 cause for me, 9.28 works very reliably.
Remember we only use the cmd 7z.exe in CMP only with 7z archives.
So if you don't use LZMA2, and one day will t7z the set, then 7z.exe is just a 'middle man'
Still I will test 9.32 thoroughly.
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 15 March 2014, 18:33
I just tried 9.32 7z UI, seems fine.
Oh, this warning ?

(http://ndsxdelta.no-ip.org/Capture_03162014_022742.PNG)

That is nothing ! You only see that when you click "Information" button.
I thought maybe 7z UI was reporting error OR (Annoying pop-up box) (Click OK) when testing archives or something.
Again Nothing, 9.32 seems fine, use it then.
2.30AM (again) zzzzzzzzz
Title: Re: 7z blockwise memory decompress
Post by: Roman on 15 March 2014, 18:47
Thanks for testing everybody....highly appreciated...looks like you get something before Easter Holiday :)
...and I'm happy that after all these years I finally got the c++ sdk part running (not so bad after all :))

One of the bigger things on the list done....next thing..hmmm..new rebuilder core....(ehe...I'd say don't expect anything before 2015....)
Title: Re: 7z blockwise memory decompress
Post by: oddi on 15 March 2014, 18:49
Oxy, about saga with 7z, me used Total Commander with two internal plugins for 7z, yesterday upgrade manually one plugin to 9.32 x64. Separated have instaled version of 9.32 x64 in win8.1 x64. Little circus :P :P :P
Btw - about my mame t7z sets, see pic: ( date column ). That is not problem, I dont know where is problem with incorrect date TC or 7z, but is not fatality.
(http://store.picbg.net/pubpic/B5/F7/c8c5801d15a1b5f7.PNG)

Roman , if u not remember - don't touch my engine.cfg  :P :P :P

Found one problem with t7z LZMA - cmp with 7z 9.32 not remove files from t7z archive. If decompress and build new 7z archive with 9.32 LZMA2 - cmp no problem with remove files inside archive.
Title: Re: 7z blockwise memory decompress
Post by: oxyandy on 16 March 2014, 09:33
Hi oddi
Quote
"Found one problem with t7z LZMA - cmp with 7z 9.32 not remove files from t7z archive."
Well, see that's important, I will confirm same issue.
Maybe have time this evening (another 3-5 hours)
Oh are your delete strings correct ?
I use
Code: [Select]
d -y -ms=off -m1=LZMA -mx0 %1 %2
___________________________________________________________________________________

Yes confirmed,
9.32 is not removing files from t7z archives..

(http://ndsxdelta.no-ip.org/confirmed.png)

Swapped back to 9.28 alpha - worked perfect. ;)
Title: Re: 7z blockwise memory decompress
Post by: Roman on 16 March 2014, 14:03
well...removal is done via calling the 7z exe with the parameters stated in the 7z tab....maybe they are not supported anymore in 7.32....
Title: Re: 7z blockwise memory decompress
Post by: Roman on 16 March 2014, 14:26
delete works fine for me with

"7-Zip [64] 9.32 alpha  Copyright (c) 1999-2013 Igor Pavlov  2013-12-01"

..and delete settings:

d -y -ms=off -mx9 %1 %2


...maybe you're using solid archives...
Title: Sorry
Post by: oxyandy on 18 March 2014, 10:20
I am guilty of not even opening CMP since Sunday evening.
When time permits, I will do some more checking.