Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Roman]

Handle with care....and on own risk....this is a TEST version...please keep this in mind....

http://mamedev.emulab.it/clrmamepro/binaries/cmp20140918.rar

besides of 32k path support (still fiddling with some things...)

misc:    Changed the way how hash collisions are handled. A hash collision happens when you got
         identically named files with different hashes within a parent/clone relationship.
         In the past cmpro either removed the parent/clone relationship completely or you were
         forced to split merged mode.
         Now, in case of a hash collision *and* only if switch to full merged sets (scanner,
         rebuilder or merger), the rom/disk names of the clones will be switched to setname\filename.
         So clone sets are stored in subfolders of the parent set. Again, only if full merged mode
         and hash collisions are active.

added:   added option (Settings screen) to allow a full merge mode which ALWAYS stores parent/clones
         the way described above, no matter if a hash collision is there or not

removed: "Possible wrong nodump definition found" cleanup step prompt. It's actually a special
         case of hash collision and so it's handled the same way as other collisions.


....I still have one open issue with Merger's nodump handling....but who uses nodumps...and who uses the merger...

[oddi]

Hello Roman , i think found few bugs with this build :
First - whats different between "normal mode" and "subfolder mode" ?
Me test first scan with "normal mode" and cmp create subfolders with roms, to here all it' s ok
but when clean cache and restart cmp - cmp remove few roms, full remove, roms not have in cmp backup folder :
first example : 18w.7z set

Also: cmp try move and rename few chds :
jojo set, chd name from dat and from Total Commander, have any character - '
 and also cmp stop here and frozen - screen 3

[Roman]

first of all... subfilders should only appear in full merged mode.

normal  mode should not create subfolders unless you got a hash collision.
subfolder mode should always store clones in sibfolders.

cleaning the cache should not play any role here

7z removes a file sounds a bit wicked. need to test that

a freeze is unlikely sounds more like a lengthY operation

again. subfolders only in full merge mode and only when hash collisions are detected.  so 99.999% of your sets should not change at all.

[oddi]

Roman, about "removed" roms: scan my full mame set with "test" version with enable "subfolders mode" in settings menu , sets is t7z/full merged mode.
When finish me back to the last official 4.015 and scan again sets , after finish rebuild from backup folder , see window:
Now me have uncompleted mame sets ))

[Cassiel]

Re 32k, so far so good. I'm currently rebuilding the entire TOSEC (main) collection uncompressed, using the standard TOSEC folder hierarchy,  (for reasons I already mentioned in other thread) so this should be a good test. I'll feedback how it goes.

Regarding \\?\, do you think it cleaner to have CMP prefix this internally to all paths by default rather than have end user specificity specify if needed? If it makes no difference I mean...

[Roman]

nah...the user has to add  \\?\ ;-) I know it's a pain in the *** but no pain, no gain....too much trouble to add it everywhere....thanks for testing

oddi, I will have a look....t7z is of course a no-go....

one thing...when you change the subfolder/normal mode, you should also trigger split/full merge once...in yesterday's build there is one refresh missing.... this is fixed in http://mamedev.emulab.it/clrmamepro/binaries/cmp20140918.rar

oddi...ok...made a test with 18wheeler sets....a full merged set got 27 files where 2 of them (epr-23298.ic22, epr-23299.ic22) are the clone roms of 18wheels and 18wheelu....
Toggled subfolder mode and running a fix scan on them changes the names of these 2 files only...18wheels/epr-23298.ic22 and 18wheelu/epr-23299.ic22)...no file loss...toggled it again, scan fix...everything back to normal

This was tested with zip and 7z....

so...for now...I can't reproduce your issue....but I'm not using t7z.....

[oddi]

Roman, today me found free little time for testing
 test with first 2 affected sets - 18w.7z and 18wheel.7z
With 18w.7z - now testing , t7z, rebuilding ( drag and drop ) and etc etc , now :
for 18w : cmp create subsubfolder 18w2 : now set is 18w.7z/18w2/18w2/rom.blabla .
with second set 18wheel.7z all it's ok
 

[Roman]

So...no issues anymore?

play a bit (with yesterday's build) and a few sets which had issues before....maybe it was related to the missing refresh of data which was fixed in yesterday's build...

[oddi]

Roman have issue !
18w.7z set :
removed from mame set and rebuild with drag and drop :
see result :



after  finish rebuild cmp report all is ok , if when start again full scan see:




and when wanna view inside 18w.7z for subfodlers and etc etc :



subfolder 18w2 it's missing

[PS] - When rebuild with drag and drop set 18w it's uncompressed , me drag folder 18w
this 2 options is ticket:

"Parse rom merge tags"
and
"subfolders mode"

Test only first affected set 18w.7z, u know result from yesterday , with many missing roms after full scan:)

[Roman]

Eh??? first screenshot shows a bad crc...then you do a rebuild?
I'm a little bit confused on the steps you're doing...


What I did: rebuilt 18wheeler parent/clone files to a new folder...full merged and subfolder mode disabled, scanned, all fine, turned on subfolder mode, scanned again and it of course asked to rename 2 files...done...triggered the mode again, scanned again, fixed names again...

so....currently I dont see
a) why you get a wrong checksum (screenshot 1)
b) why it wants to add a file (screenshot 2 + 3)

so again: which MAME version are you using?

[oddi]

first screen - removed "18w.7z" not "18wheelr.7z" from mame set, when scan mame set cmp report for 2 missing sets.
after rebuild cmp report this , if clean cache and scan again cmp ask:
----------------
Set:   18 Wheeler (set 1)
Name:   18w
File:   G:\MAME\Mame - ROMs\18w\18w_b1.rom1

Do you want to add the missing ROM

and this:

Set:   18 Wheeler (set 2)
Name:   18w2
File:   G:\MAME\Mame - ROMs\18w2\18w_b1.rom1

Do you want to add the missing ROM?

and now before that 18w set have 5 roms !!!
after scan think have 4 roms parent +1 in subfolder "18w2"
but atm i view in archive only 4 parrent , 1 rom it's missing, missing subfolder 18w2  ( the last screenshot)
i check which rom s missing:
this:

btw - test with version:
http://mamedev.emulab.it/clrmamepro/binaries/cmp20140918.rar

used dat from mame svn: M.A.M.E. v0.154 (Sep 17 2014)svn32155.dat
dat upload here:
http://www.speedyshare.com/BP9a5/M.A.M.E.-v0.154-Sep-17-2014-svn32155.7z

[oddi]

btw - now test cmp with MESS bioses sets - cmp work perfect !!! all roms it's ok with subfolders and etc , phhhhhhhhfffffffffff but mame........hard job,
sorry Roman

[PS] - O no ! Now cmp work with mame sets too, dont know but cmp now move roms to backup folder , i'm rebuil from backup folder and cmp create 18w2 subfolder !!!!!
mad already! where I'm wrong,

first full clean cache and rescan full again , again problem with this set ! 18w.7z :
again bad checksum
 and rom is missing, not in backup folder too , away

[oddi]

Chaos is big , now after full cache cleanup , mess bioses is too chaos , after first scan cmp create subfolders, after clean cache and scan again cmp move all roms to parrent and saty empty subfolders !!
ohh, very tired with cmp , think stop for this evening.

[Roman]

ah.....unpacked sets....yeah...there is a problem...because it tries to remove the subfolder ;-)
...and yesterdays build also got an issue when switching the mode again and again and again...it always added a subfolder level..

latter issue is fixed..now I look at the unpacked folder issue....thanks oddie
(rar/7z/zip files are NOT affected though)

in other words....no need to do more testing (actually you should NOT use it with uncompressed sets at the moment).....until I release an update this weekend.

I've removed the test binary for now....please do not use it anymore....

[Cassiel]

I'm currently rebuilding the entire TOSEC (main) collection uncompressed, using the standard TOSEC folder hierarchy, (for reasons I already mentioned in other thread) so this should be a good test. I'll feedback how it goes.

Full TOSEC (main) collection rebuild completed.

Using WIP 20140917 exe, rebuilding to uncompressed, rebuilding from uncompressed/RAR v5 mix of files.

No major problems experienced, except:
- CMP crashed on 'C64 - Games - [D64]' DAT when adding one missed file from previously rebuild run (but then this DAT does include around 100k small files/folders so not that surprised, might not have even been CMP fault)
- CMP *sometimes* left files in backup folder that it shouldn’t. These generally appear to be >260 path ones but not always (I think) and definitely not all of the >260 rebuilt. Was only a handful of files and could not notice any specific pattern. Stupidly I wasn’t paying enough attention at the time to be make a note of which files and 'how' I was rebuilding them.

[Roman]

do NOT use any build which got the subfolder/hashcollision stuff enabled...
Do you have any (small) repeatable scenario for your remarks?

[Cassiel]

do NOT use any build which got the subfolder/hashcollision stuff enabled...

I read above, but I'm not using any DATs with any kind off parent/clone relationship. I had to use this new WIP build to achieve what I was after doing...   

Do you have any (small) repeatable scenario for your remarks?

Scenario 1, not really unless you have a go with 100k commodore images. I'm happy to provide everything you need if you want though.
Scenario 2, like I said was very stupid by emptying temp folder. Realized 0.5 sec later that should have kept a note of the handful of files, but wasn't thinking at the time.   

[Cassiel]

RE 2, I can have another play in >5 hours and see if anything similar happens?

(waiting on something to finish using these files)

[Roman]

so if you're wondering what's the status...well...actually I got only a very limited amount of time to concentrate on cmpro....regarding the subfolder / hashcollision thingie, there are some scanner things I need to handle better.
While packed sets are rather harmless, the problem comes when you handle decompressed sets....and if you now got subfolders (due to hash collisions or chosen new mode), there are some points here and there where the scanner needs some updating...
Especially if you switch the mode back to split merged, such subfolders become unneeded....but you cannot simply remove them because the files in them needs to be moved to their correct place (the clone folder) first before you remove the unneeded folder...so that's what I'm currently looking into....

Guess you need to be a bit more patient....

[B2K24]

Especially if you switch the mode back to split merged, such subfolders become unneeded....but you cannot simply remove them because the files in them needs to be moved to their correct place (the clone folder) first before you remove the unneeded folder...so that's what I'm currently looking into....

Guess you need to be a bit more patient....

Are you saying people set rom path to a fully merged set and scan it as split merged?
That sounds like absolute stupidity.

It's far easier to just use the rebuilder and rebuild a split merged set sourced from merged files in an empty destination directory.