Topic: 7z blockwise memory decompress (Read 109006 times)

oxyandy · « **Reply #20 on:** 12 March 2014, 14:21 »

Regarding the empty folders....I wonder how old cmpro reacts....
Yeah, guess what old 4.012b CMP isn't seeing/removing them either.
I 'think' I have noticed that maybe since a few CMP revisions that "Rebuilder"
while removing files housed in a sub-folder in rar, that an empty folder remains after file is removed from it..
Did CMP previously remove the file & folder if removing the rebuilt file, left the folder empty ?
I guess I had better check. But something does 'feel' different there.

- test archive works but may return a wrong information
Seemed to work on OK archives, but test on corrupt files is showing them as OK too.
- corrupt 7z files are not always marked as being corrupt (-> message in warnings window or something)
Corrupt archives are always testing as OK, they should fail 'Test'.

- empty folders are not reported as unneeded
Yeah, but this behaviour is the same on 4.012b 'Official Release' - I just checked

oxyandy · « **Reply #21 on:** 12 March 2014, 14:53 »

Gee, just tried 4.08b - it is not seeing the 'Not in DAT', empty folders in the 7z archive either.

Alright, converted my 7z archive to zip, cleared cache,
'New Scan' now I get prompted to Remove the not in dat, empty folders.

Summary "Scanner" with 4.08b:
1. ZIP empty, not in dat folders, are seen and removed.
2. RAR empty, not in dat folders, are seen and removed. (Inc. those double pesky folder entries WinRAR exe makes)
3. 7z empty, not in dat folders, are NOT even seen, so of course, no prompt to remove them..

Guess I found an old bug !
haha

See something changed with WinRAR!
Summary "Scanner" with 4.012b 'Official Release':
1. ZIP empty, not in dat folders, are seen and removed.
2. RAR empty, not in dat folders, are NOT even seen, so of course, no prompt to remove them..
3. 7z empty, not in dat folders, are NOT even seen, so of course, no prompt to remove them..

Starshadow · « **Reply #22 on:** 12 March 2014, 16:27 »

Quote from: oxyandy on 12 March 2014, 13:12

Starshadow, what about without - 'Test archive (decompress to memory) (Scanner only)' ticked ?
(or in Scanner - 'Hash & CHD' - "Decompress Rom & CRC/SHA Check" checked.)
Just a regular 'New Scan' ?

New Scan without either option checked works just fine. The "Decompress Rom & Check..." options appear to work fine as well.

Roman · « **Reply #23 on:** 12 March 2014, 19:15 »

ah ok...the detection of obsolete subfolders is limited internally to zip....since removing them only works safely in zip...for rar/7z it might remove more than just the entry for the folder... it was disabled with 4.011a

so this build:

http://mamedev.emulab.it/clrmamepro/binaries/cmp20140312.rar

should fix your other reported things...

I may have a look at the empty folder things in the next days...

oxyandy · « **Reply #24 on:** 13 March 2014, 01:18 »

Hi Roman,
I am still playing with the new build.
First impressions are it seems you have resolved everything reported..
Corrupt archives are shown now (both test & decompress.........), via "Scanner"
but still no msg when a corrupt 7z is found in source path (No big deal anyway) via "Rebuilder"
I could suggest a few 'tweaks' - but also feeling lazy at this point..
I will play further on & off, across the day - if anything drastic is noticed, I will post.
Otherwise, it will be a case of no news, is good news.

7z Over 4 Gigabyte Archives .. I wont be testing.. so wait for feedback from someone for those..

oxyandy · « **Reply #25 on:** 13 March 2014, 08:53 »

Hello from "64 Bit Land",

I am seeing cmpro64 not working as it should too.
Wont Rebuild to 7z or Scan 7z...
Sorry, no time left, work & life calls

EDIT: I had almost no time at all to do much testing with either the 'new' 32 or 64 builds today..
Only a few minutes with either.

Roman · « **Reply #26 on:** 13 March 2014, 11:03 »

hmm...sounds weird...have no issue...will check later this evening...

Starshadow · « **Reply #27 on:** 13 March 2014, 12:00 »

cmp20140312 64-bit solved the issue I reported when running a scan with the 'Test Archives' option turned on, but as oxyandy alluded to, now when running a New Scan with the 'Decompress ROM & Check...' options enabled, I get 'error while unpacking 7z file:' and 'Archive Header CRC32 doesn't match CRC32 of decompressed data for file:' for every archive. Rebuilding to/from 7z seems to be ok to me though, he'll have to provide more info on what he's seeing there.

Roman · « **Reply #28 on:** 13 March 2014, 12:03 »

yeah yeah....somthing's spooky there....will check it tonight...thanks for the support

oxyandy · « **Reply #29 on:** 13 March 2014, 12:36 »

Hi Starshadow,
Again I admit both attempts I made with 'today's new builds' were both very hurried..
I use the 7z.exe from 9.28 alpha (on 32 bit) cause that version seemed the best when I made lots of tests long ago.
The x64 test a few hours ago was with 7z.exe from 9.25 x64, cause that was all I could find quickly..
What cmd version of 7z.exe are you using ?
I have promised Roman in about another 40 minutes I will confirm a more serious test with x64..

Yeah the new version CMP32 "Decompress & check..." is not working as it should, either..
It worked on the previous version.. (well sort of)
But 'test' in compressor setting now seems to work ok.. confirmed.

PS. I have since tracked down x64 7z.exe from 9.28 alpha, so I will try with that..

oddi · « **Reply #30 on:** 13 March 2014, 13:17 »

Hello guys, i'm happy see hard job between cmp and 7z.
Me test cmp x64 and 7z 9.32 x64 ( replace 9.28alpha ) for test.
Old bug when cmp not rename roms inside archive is fixed.
Btw - very very strange problem :
first 2 sets from mame: 005.7z and 10yard.7z :
If rename 005.7z to any name, example: 00523blabla.7z , cmp scan and ask for rename
but rename 10yard.7z to any name : example 10yardblabla.7z , cmp wanna remove file !?!
if me give permission for remove after that no problem with restore with rebuild.
Other little problem: maybe this problem only from 7z side :
if test any t7z archive with 7z 9.32 x64:
7z report: "There are data after the end of archive"

oxyandy · « **Reply #31 on:** 13 March 2014, 13:58 »

Hi Roman,
x64 CMP seems fine now, only thing it is doing which it shouldn't is
In "Scanner" "Hash & CHD" "Decompress ROM & check...." is producing a 7z error msg..
(Same with CMP32)
AH, now I know why I didn't see this earlier and maybe why you haven't seen it too !
I was rushing so didn't wait till (Files to go =1)

OK, I click "New Scan" on a folder with a single 7z archive, this archive has 50 internal files.
Files to go counts down from 50,
NO error message is produced.
49, 48, 47, 46 ..all the way down to... 2 NO WARNING WINDOW !
Then once "Files to go", reaches ONE,
THEN the warning window appears.
"Files to go", STAYS as ONE,
Then it one by one produces an error message for each internal file.

Hi Oddi,
Don't use GUI of 7z 9.32 x64 and you wont get "There are data after the end of archive"
These are t7z (torrent 7 zip) archives..
Use an older GUI, not sure which version, try a different 7z GUI version and see,
then no more "There are data after the end of archive" message

Oddi you say cmd line EXE of 7z v9.32 is working OK with CMP ?
Ok, I will give it a try, thanks

Roman · « **Reply #32 on:** 13 March 2014, 18:36 »

ah silly me...pretty obvious error...fixed now...I will now check the (unneeded) folder stuff so this might get reenabled....expect a new test version later tonight....

oxyandy · « **Reply #33 on:** 13 March 2014, 18:40 »

Ah good,
so you made sense of my post, haha
2.39 AM my time, being ordered to bed by the boss..
Great work, yet again !
Thx Roman.

Roman · « **Reply #34 on:** 13 March 2014, 21:38 »

ok here we go...this should fix the problems

http://mamedev.emulab.it/clrmamepro/binaries/cmp20140313.rar

...plus I've reenabled the detection of obsolete folder entries in rar/7z and made the removal of such entries via external packers more safe (in the past the external packers killed the folder...no matter what's inside

oxyandy · « **Reply #35 on:** 14 March 2014, 02:35 »

I am 'Awake'.
You think it's bug free this time ?
I'll keep testing !
I'll test everything..
From..
Source file 7z (internal files total 50) 25 match the DAT.
And Rebuild it..
will remaining source archive be not corrupt with it's remaining 25 Files ?
Destination file ok too ?
To..
Destination files, with sub-folders 'as defined in the DAT',
then move the files around with manual Notepad++ DAT edits etc, etc
Test the 'Empty Folders' & everything else I can think of.

Right now, using CMP32, I see yesterday's bugs are indeed gone !
This 7z update is letting me do things with 7z archives that previously would fail on a 32bit OS
It really is a wonderful advancement !

IF you have 'repaired' the 'Empty Folder' removal..
Then only with 'this' release, will CMP be truly honouring the DAT in relation to DAT/Archive Sub-Folder entries.
Across any archive type, ZIP, 7z & Rar - a First for CMP!
Honouring the DAT faithfully is, of course, what every 'ROM Manager' should do.
So I think this is SUPER important update !

Starshadow please,
Some feedback if you will, for the massive over 4Gb archives, even with the 32Bit CMP if possible.
I just don't have any datted 4Gb 7z's in any of my collection.

oddi · « **Reply #36 on:** 14 March 2014, 03:00 »

CMP x64 and 7zip 9.32 - not rebuild roms with drag and drop, cmp report error. Back to 9.28 - all is ok.

oxyandy · « **Reply #37 on:** 14 March 2014, 03:17 »

Thanks for feedback Oddi !
Right now I am making use of 7z.exe (7z.exe command line v9.32 -32 bit)
It seems to be working.. (But for sure I will keep this in my thoughts)

Roman,
I found a bug, where CMP broke the 7z archive with duplicate filenames. (Better feedback later)
As I warned, I will test everything, my goal - break CMP's working operations.
I will produce 2 DAT files & sample TEST 7z archives for you to REPEAT the operation, later..
Step by Step testing first.

CMP is great though, it is seeing the dupe entry & reporting it.

Of course now the error is there, no further processing can be made on this archive

Dumped my broken archive.
Started fresh,
Maybe I am seeing the same as you here Oddi ? (7z.exe command line v9.32 -32 bit)

I'll roll-back to v9.28, see if error goes.
The Result: Yes, Rebuilder message gone..
I will swap back to v9.32 - confirm it returns
The Result: No, same operation as before, this time it continues. Damn
Test continues on v9.32

Roman · « **Reply #38 on:** 14 March 2014, 07:59 »

so we have an issue with 7z.exe 9.32...and one with double file entries in a 7z file (what the heck..this should not exist anyway...)....right?

Roman · « **Reply #39 on:** 14 March 2014, 08:37 »

I guess I can look at it in 10 to 12 hours....

News:

Author Topic: 7z blockwise memory decompress (Read 109006 times)