EMULAB Forum
clrmamepro [English] => clrmame Discussion => Topic started by: Roman on 26 October 2023, 18:05
-
Ok, here we go....a test version for some delta chd features.
Disclaimer: This is not an official release, it's just a 64bit cmpro exe file which can replace yours. Do it on your own risk. Any chd convert/revert operation, you do it on your own risk.
What are the delta chd features (screenshots here: https://www.emulab.it/forum/index.php?topic=9148.msg25902#msg25902) ?
In Settings->Compressor...->CHDMan, you find a new section "Delta CHDs". It also holds two new command line boxes for converting and reverting. They are filled with the default copy operations, don't change them.
The section holds options where you can decide if cmpro
- should warn you about chds which can be converted to deltas
- chds which are deltas
- or you don't want to see any warnings at all (default).
There is also a checkbox (ticked by default) which enables backup when a convert/revert operation actually takes place.
After running a (new) scan and having a warning option enabled, you might see warnings in the warnings window below the scanner. It can list something like:
- Non-delta chd: some.chd Convertable via someother.chd ( showed warning about non-delta chds )
- Delta chd: some.chd Revertable via someother.chd (showed warning about delta chds)
- Bad parent delta chd: some.chd Revertable via someother.chd (error: parents can't be deltas)
- Bad clone delta chd: some.chd Revertable via someother.chd (error: the delta clone uses a different chd than the current parent)
- Can't convert/revert some.chd. Can't find a non delta CHD with SHA1: somesha1 (error: parent is not available)
If one or more of the messages are shown, you can use the popup contextmenu on one of the lines (you can also select multiple) and select one of the delta chd options in the menu. You can convert or revert all/selected. Depending on the selection you see either the one or the other or both options...
When selecting convert or revert, chdman is called and run in the background. If something went wrong or your used ctrl-c to exit chdman and you got multiple files in the queue, you will be asked if you want to continue.
If you ticked the backup option (you should), delta and absolute chds are copied to your backup folder. For this, subfolders are created in there _chd_abs_ and chd_delta_.
Known things:
- delta chds are always smaller? No, there are cases, especially when it comes to examples where the parent chd is a v4 baddump and the clone is a v5 chd, the delta version is slightly bigger
- when a chd is converted and reverted, I get the original file back? No, not necessarily. There are chds which were created with an older version of chdman and converting/reverting them with the current one ends up with a slightly smaller file. This is most likely based on the fact that the newer chdman uses a newer version of lzma which is responsible for the size loss. However, the raw data and its sha1 will be the same.
- when thinking about reverting, cmpro's unneeded/set check helps to find parent chds which might be located elsewhere or named incorrectly
- Can I convert all clone chds? If there is a belonging parent chd, yes (when CHD version is >= 3). Belonging? Well, there are sets which have more than one chd. To identify a belonging chd, the chd's "region" (and for software lists, the "part") information is used for a match. If a chd is not listed with a region/part, cmpro won't use it for any delta operations.
Well..happy testing...and don't forget: it's your fault when something goes wrong ;-)
Don't know if it's worth all the trouble....in times of multi terabyte HDs.....but the warnings might be useful.
-
Hi Roman, not working ! :)
I try all - with delta, miss delta, bad delta ...and more delta warnings...and nothing.
Cmpro completed scan without warnings :)
All is fine, my mistake :)
-
D'oh, I was watching the other thread and missed this! Downloaded the new release build and am testing. Thanks Roman!
-
Ok, I just hit a few errors 'an error happened during working on a CHD, do you want to stop yes or no'. If I choose no it prompts me again when it finds the next one. It would be nice to say 'no to all' and let it run without being touched.
The CHDs it choked on were ddr4mj, ddr4msj, dmx so far. Is there logging code to see what happened? When I run a regular scan after those errors cmpro tells me those CHDs are 'unneeded' now. Maybe they were originally damaged but unnoticed and this deeper dive is calling them out.
-
Here are some of the delta CHDs that get created but are flagged as wrong, and then missing if I remove them:
Dance Dance Revolution 4th Mix (G*A33 VER. JAA) [folder: ddr4mj]
missing chd: a33jaa02.chd [chd-md5: N/A] [chd-sha1: 9d9fb5e65f1532f358e9c273c56d11389d11fd79]
Dance Dance Revolution 4th Mix Solo (G*A33 VER. JBA) [folder: ddr4msj]
missing chd: a33jba02.chd [chd-md5: N/A] [chd-sha1: 9d9fb5e65f1532f358e9c273c56d11389d11fd79]
Dance Maniax (G*874 VER. JAA) [folder: dmx]
missing chd: 874jaa.chd [chd-md5: N/A] [chd-sha1: 3338a784efdca4f8bdcc83d2c9a6bbe7f7046d5c]
Dancing Stage featuring Dreams Come True (GC910 VER. JAA) [folder: dsfdcta]
missing chd: 910jaa02.chd [chd-md5: N/A] [chd-sha1: 70851c383e3876c4a697a99706fbaae2dafcb0e0]
Bass Angler 2 (GE865 VER. JAA) [folder: bassang2]
missing chd: 865jaa02.chd [chd-md5: N/A] [chd-sha1: b98d9aa54f13aa73bea580d6494cb6a7f3217be3]
Guitar Freaks 3rd Mix - security cassette versionup (949JAZ02) [folder: gtfrk3mb]
missing chd: 949jaz02.chd [chd-md5: N/A] [chd-sha1: b0c786ba420a34fcbd16bc36a137f6ae87b7dfa8]
Guitar Freaks 8th Mix (G*C08 VER. JAA) [folder: gtrfrk8ma]
missing chd: c08jaa02.chd [chd-md5: N/A] [chd-sha1: 7a1d97f74ec4d643ff7d3981d66b551cbf9e57f0]
Quizard (v1.2, German, i8751 DE 11 D3) [folder: quizard_12]
missing chd: quizard12.chd [chd-md5: N/A] [chd-sha1: 6e41683b96b74e903040842aeb18437ad7813c82]
Quizard (v1.7, German, i8751 DE 11 D3) [folder: quizard_17]
missing chd: quizard17.chd [chd-md5: N/A] [chd-sha1: 4bd698f076505b4e17be978481bce027eb47123b]
Quizard (v1.2, Italian, i8751 IT 11 I2) [folder: quizardi_12]
missing chd: quizard12.chd [chd-md5: N/A] [chd-sha1: 6e41683b96b74e903040842aeb18437ad7813c82]
Quizard (v1.7, Italian, i8751 IT 11 I2) [folder: quizardi_17]
missing chd: quizard17.chd [chd-md5: N/A] [chd-sha1: 4bd698f076505b4e17be978481bce027eb47123b]
Quizard 2 (v2.2, German, i8751 DN 122 D3) [folder: quizard2_22]
missing chd: quizard22.chd [chd-md5: N/A] [chd-sha1: 03c8fdcf27ead6e221691111e8c679b551099543]
Quizard 3 (v3.2, German, i8751 DE 132 D3) [folder: quizard3_32]
missing chd: quizard32.chd [chd-md5: N/A] [chd-sha1: 31e9fa2169aa44d799c37170b238134ab738e1a1]
Quizard 4 Rainbow (v4.0, German, i8751 DE 142 D3) [folder: quizard4_40]
missing chd: quizard4r40.chd [chd-md5: N/A] [chd-sha1: 288cc37a994e4f1cbd47aa8c92342879c6fc0b87]
Quizard 4 Rainbow (v4.1, German, i8751 DE 142 D3) [folder: quizard4_41]
missing chd: quizard4r41.chd [chd-md5: N/A] [chd-sha1: 2c0484c6545aac8e00b318328c6edce6f5dde43d]
-
Latest batch:
Error while converting chd: 'C:\O\Games\MAME\ROMs\ddrja\845jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\ddru\845uaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\dstagea\845uaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\gtrfrksac\886__c02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\gtrfrksc\886__c02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\gtrfrksjc\886__c02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\gtrfrksuc\886__c02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\susume\gv027j1.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\netchu02b\npy1cd0b.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn10m\d40jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn2m\912jab02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn2mpu\912jab02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn3m\a23jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\pcnfrk3mk\a23kaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn4m\a25jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn4m\a25jba02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn5m\b05jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn6m\b16jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn7ma\c07jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn9m\d09jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\drmn9ma\d09jaa02.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\redearthr1\cap-wzd-3.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\warzardr1\cap-wzd-3.chd'
Error while converting chd: 'C:\O\Games\MAME\ROMs\totlvicj\639jad01.chd'
These error out of Chdman when manually run too, just a percentage and then a quit to command line. The resulting malformed delta CHD is then flagged as wrong for clrmame removal on subsequent scan.
-
Something new from delta chds for a moment. Clrmame is quitting when it's trying to move freestanding *.chd files into a subdirectory structure.
I decided to convert from split to merged storage. I moved my CHDs out of their \roms\blah directories and put the raw *.chd files in the \roms directory.
In the past when I've done this and run a scan, clrmame will find them and move them to their created respective subdirectories. Instead of doing that now it just quits out of clrmame.
The scan is also not finding the the CHDs in the add-folder. Rebuild does not move the CHDs over from the add-folder either. I purged my clrmame setup and re-downloaded and started from scratch just in case it was an errant setting, but same problem.
-
Something new from delta chds for a moment. Clrmame is quitting when it's trying to move freestanding *.chd files into a subdirectory structure.
I decided to convert from split to merged storage. I moved my CHDs out of their \roms\blah directories and put the raw *.chd files in the \roms directory.
In the past when I've done this and run a scan, clrmame will find them and move them to their created respective subdirectories. Instead of doing that now it just quits out of clrmame.
The scan is also not finding the the CHDs in the add-folder. Rebuild does not move the CHDs over from the add-folder either. I purged my clrmame setup and re-downloaded and started from scratch just in case it was an errant setting, but same problem.
Was about to post the same thing!
-
I went back to 4.047a and it worked again. So it's a regression from there. :)
-
wow..lots of information...I try to work on them as quick as possible...
regarding the convert fails on some chds, maybe it's related to version differences....there are some parent v4 chds in MAME collections which are flagged as baddump where the clones are v5 chds....
Checking the rest hopefully soon....
-
the crash / exit is fixed.....Rebuilder does not work with chds, neither does fix-missing on addfiles if I'm not mistaken...
....so I will look into the drmn10m etc chds what's going on there.....if manual converting doesn't work either it's something related to the chds (maybe version mismatch).....stay tuned
-
Results with new .exe. There are a bunch of CHDs being flagged as delta candidates but they're the same CHD (parent and child). When you do the conversion on them it outputs a malformed, small CHD.
Non-delta chd: 'C:\O\Games\MAME\ROMs\area51\area51.chd '. Convertable via 'C:\O\Games\MAME\ROMs\area51\area51.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\area51mx\area51mx.chd '. Convertable via 'C:\O\Games\MAME\ROMs\area51mx\area51mx.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx\863hdda01.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx\863hdda01.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx\gq863a04.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx\gq863a04.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx3\992hdda01.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx3\992hdda01.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx3\gc992-jaa04.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx3\gc992-jaa04.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx3\992hdda01.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx3\992hdda01.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx3\gc992-jaa04.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx3\gc992-jaa04.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx6\b4ujaa02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx6\b4ujaa02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx6\b4ujaa03.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx6\b4ujaa03.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx7\b44jaa02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx7\b44jaa02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidx7\b44jaa03.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx7\b44jaa03.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidxc\863hdda01.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx\863hdda01.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidxc\gq863a04.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidx\gq863a04.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidxs\983hdda01.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidxc2\983hdda01.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\bmiidxs\gc983a04.chd '. Convertable via 'C:\O\Games\MAME\ROMs\bmiidxc2\gc983a04.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\clubkcyc\cycraft_clubkart_smcp_v1.65.chd '. Convertable via 'C:\O\Games\MAME\ROMs\clubkcyc\cycraft_clubkart_smcp_v1.65.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\dkbanana\dkbanana.chd '. Convertable via 'C:\O\Games\MAME\ROMs\dkbanana\dkbanana.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\esh\esh_ver2_en.chd '. Convertable via 'C:\O\Games\MAME\ROMs\esh\esh_ver2_en.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\gtrfrks\886__d02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\gtrfrks\886__d02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\inidv3cy\initiald3_smcp_v1.66.chd '. Convertable via 'C:\O\Games\MAME\ROMs\inidv3cy\initiald3_smcp_v1.66.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\jojo\cap-jjk-3.chd '. Convertable via 'C:\O\Games\MAME\ROMs\jojo\cap-jjk-3.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\kinst2\kinst2.chd '. Convertable via 'C:\O\Games\MAME\ROMs\kinst2\kinst2.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\lacrazyc\gv027j1.chd '. Convertable via 'C:\O\Games\MAME\ROMs\lacrazyc\gv027-a1.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\mamboagg\a40jab02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\mamboagg\a40jab02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\pcnfrk\881xxb02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\pcnfrk\881xxb02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\pcnfrk9m\d09jca02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\pcnfrk9m\d09aba02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\photoply2k1it\nokia_snake_ii_service_release_update_2001.chd '. Convertable via 'C:\O\Games\MAME\ROMs\photoply2k1it\nokia_snake_ii_service_release_update_2001.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\photoply2k1it\update_2001_cd1.chd '. Convertable via 'C:\O\Games\MAME\ROMs\photoply2k1it\update_2001_cd1.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\photoply2k1it\update_2001_cd2.chd '. Convertable via 'C:\O\Games\MAME\ROMs\photoply2k1it\update_2001_cd2.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\popn1\803_ta_hdd.chd '. Convertable via 'C:\O\Games\MAME\ROMs\popn1\803_ta_hdd.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\popnanm\gq987jaa01.chd '. Convertable via 'C:\O\Games\MAME\ROMs\popnanm\gq987jaa01.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\popnanm\gq987jaa02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\popnanm\gq987jaa02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\popnanm2\gea02jaa01.chd '. Convertable via 'C:\O\Games\MAME\ROMs\popnanm2\gea02jaa01.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\popnanm2\gea02jaa02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\popnanm2\gea02jaa02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\popnmt\976jaa02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\popnmt\976jaa02.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\redearth\cap-wzd-5.chd '. Convertable via 'C:\O\Games\MAME\ROMs\redearth\cap-wzd-5.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\sfiii\cap-sf3-3.chd '. Convertable via 'C:\O\Games\MAME\ROMs\sfiii\cap-sf3-3.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\sfiii2\cap-3ga000.chd '. Convertable via 'C:\O\Games\MAME\ROMs\sfiii2\cap-3ga000.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\sfiii3\cap-33s-2.chd '. Convertable via 'C:\O\Games\MAME\ROMs\sfiii3\cap-33s-2.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\timecrs4\tsf1-ha.chd '. Convertable via 'C:\O\Games\MAME\ROMs\timecrs4\tsf1-ha.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\timetrv\timetrv.chd '. Convertable via 'C:\O\Games\MAME\ROMs\timetrv\timetrv.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\ultarctc\uarctict.chd '. Convertable via 'C:\O\Games\MAME\ROMs\ultarctc\uarctict.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\vjslap\vj_ver1.chd '. Convertable via 'C:\O\Games\MAME\ROMs\vjslap\vj_ver1.chd'
Non-delta chd: 'C:\O\Games\MAME\ROMs\wcombat\c22d02.chd '. Convertable via 'C:\O\Games\MAME\ROMs\wcombat\c22d02.chd'
-
erm...weird..never seen that...which merge mode do you use? how are parent/clone (area51 to take one example) stored (Well I see area51.chd but real clone?)
-
I have just about gotten everything back together in merged mode which I haven't used in like 15 years, laugh. It's tough because the CHD move during scan won't proceed if there is a same named CHD in the final destination even if it's the 'wrong' one.
Due to merged mode I have roms\area51\*.chds. In the area51 directory are the two files:
"C:\O\Games\MAME\ROMs\area51\area51t.chd"
"C:\O\Games\MAME\ROMs\area51\area51.chd"
The area51t.chd was converted yesterday and is a delta.
-
ok...found it....I shouldn't work on weekends...
-
Appreciate all you do w/ this vital app Roman, as always. :)
-
hehe....you've got a private message...
-
Replied. Cheers -
-
I think I miss something here. I don't see any warnings about delta after full scan even if I choose option about showing delta. Do I have to full scan with crc checking of chds ?
-
Since nobody I know of uses delta chd, you most likely don't see any warnings about delta chds....
So I assume you don't have any delta chds....if you select "warn about non delta" you should see a warning for nearly all of your chds. The warnings appear in the warnings window below the scanner.
-
I didn't have any, so far
Window was behind scanner window ;D
-
Ok, I've converted couple of chds into delta but right now scanner (full with chd integrity check) shows those deltas as entegrity check failed :o
-
Well yes, since cmpro does a "chdman verify -i %1" command which would fail on deltas....it would require "chdman verify -i %1 -ip %2" where %2 gets replaced with the parent chd filename...but that's currently not supported (but on the to do list).
So...for delta the deep check won't work....but as I said: currently nobody is using delta chds since there are some open issues with them on chdman side, see https://github.com/mamedev/mame/issues/11675
-
Yeah, I've seen this issue. For now feature in test but I was curious how much it could save disk space and time for full test chds. Guess we have to wait a bit before full switch to deltas :)
-
It's about 300GB savings over the mame chds (non-software list).
-
That's lot of disk space :D
-
If you can't wait till next year and really need deep/full integrity check support for delta chds, you can download this build:
https://mamedev.emulab.it/clrmamepro/binaries/cmpro_deltaverify.7z
and unpack/replace you cmpro64.exe with it
-
I did some tests and appears to be working correctly but of course not all chds are converted.
-
I converted the CHDs to deltas, the only errors I found were these:
-
well....guess we all have to wait till MAME team fixes the chdman issue sooner or later
https://github.com/mamedev/mame/issues/11675
-
Well yes, since cmpro does a "chdman verify -i %1" command which would fail on deltas....it would require "chdman verify -i %1 -ip %2" where %2 gets replaced with the parent chd filename...but that's currently not supported (but on the to do list).
So...for delta the deep check won't work....but as I said: currently nobody is using delta chds since there are some open issues with them on chdman side, see https://github.com/mamedev/mame/issues/11675
Hi Roman, in the course of investigating some of the malformed delta chd creations, I tried to use cmpro to do validity checks on my existing chds and hit this problem above w/ a new commandline param needed for delta chd verification. Is that feature mature enough to release into a full build at this point? Thanks -
-
You need to use https://mamedev.emulab.it/clrmamepro/binaries/cmpro_deltaverify.7z
which is rock stable :)
Real life job currently takes all my time.....