Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[john iv]

Hi Roman, I assume you've seen:
https://github.com/mamedev/mame/commit/d1172bf710f2a7b1777ed7687f5ba65748d90ee9

This could lead to significant savings on child CHDs becoming deltas of their parents. Have you got your mind around how to support it? I'm not sure how to produce one of these deltas, but a guide might be nice for the potential space savings.

[Roman]

Well, I think regarding auditing nothing much will change. I haven't looked at the changes yet, but I assume that

a) the hash / name values of the disk entries in -listxml output doesn't change
b) the chd header will get some additional flag that shows that it is a delta chd

cmpro reads only the chd header and compares the hashes against the hashes in the xml output ...so that will still work....a deep check is done via chdman anywhow...so I think that will also still work since they probably update chdman's test routine...

....so in the end it might be only something like "couldn't cmpro show me that I am able to use a delta chd instead of a full one" request....

but hey...I might be wrong...time will tell....thanks for the headup

[john iv]

I could see people abandoning the child .chds and just keeping the deltas, so was curious if you'd have to account for that.  Guess we'll see!

[Update] I just converted gtfore04a and it went from 1GB to a delta of 90MB.  Clrmame scan flagged the chd 'missing' though it was in the right place w/ the right name, fwiw.

[Update2] Roman, it would be cool if you could use Clrmame or one of your external tools to find eligable CHDs and prompt to create their deltas all in one pass.

chdman copy -i sf2049tea.chd -op sf2049te.chd -o sf2049tea-delta.chd
mv sf2049tea.chd sf2049tea-abs.chd
mv sf2049tea-delta.chd sf2049tea.chd

[Roman]

Yeah yeah keeping an old man busy....

Thanks for the testing and information....First thing I will do is of course checking why a delta is not recognized when it got the correct name in the correct place...most likely because the chd header changed a bit...

"fixing" or better "auto-create a delta" out of a found non-delta clone....hmm....will see if this is easy to add (well it is...but I have very limited time).

Question is if (or if it makes sense that) each and every clone chd can be converted to a delta compared to its parent. Haven't checked yet but I'm sure that there are sets with multiple chds and maybe a clone of it also has multiple ones...question is how to find the correct parent chd to the clone chd in such a multi-case.   ....most likely if the region entries match...like "region="pci:05.0:ide:0:hdd"" ...


to sum it up: I try my best to update cmpro as quick as possible....

[Roman]

ok...attached a quick version so that cmpro works again....can anyone give it a try for delta chds?

Thinking about

a) an optional warning if found clone disks could get replaced by a delta
b) maybe a standalone tool for converting....(guess that can be achieved quicker than adding some conversion thing into old cmpro..)

but don't count on a) and b) in the very near future....

[john iv]

Roman, the new .exe worked.  It didn't flag the four CHDs I've converted.  Good job.

[Roman]

ok..let's see what I got....and what needs to be done...

Already added the ability to:


- show an error entry (scanner, tree view) if a parent chd is a delta chd (parent chds can't be deltas, this can happen when MAME changed parent/clone relationships)
- show an error entry (scanner, tree view) if a clone chd is a delta chd but the parent sha1 stored in it doesn't match the current parent chd (as above, maybe some parent/clone relationships changed)

- optionally show a warning about a clone chd which can be a delta but isn't (warning window below scanner) (default: false)
- optionally auto converting a clone to a delta via chdman call (similar to do deep chd checks), see attached screenshot  (default: false)
- optionally doing a backup of the original clone chd before it gets converted to a delta  (default: false, gets auto-enabled with enabling converting option, but you can disable it afterwards)


To do:
- if the upper mentioned errors appear, you would need to revert the delta to the original image. For that, the right chd needs to be found first (could be outside of the current parent/clone relationship)
- software list chds are not yet supported. Currently a possible delta-conversion is detected if the parent/clone disks are available AND they share the same "region" information. In software lists you have "parts"...and I need to do some more investigation on them...if it's safe to do a match by using this information.

Generally, converting the chds takes a lot of time...plus the checks in cmpro are purely experimental for now.....Keep you updated.

[john iv]

Looking good, see you have been interacting w/ Vas and getting good info on mameworld.

[Roman]

yes….need to check how to revert a chd and I still think about the way a conversion makes sense….just having that option enabled can cause a long long scan but currently I dont have a good idea about a different way….I mean if users enable the option they know what is coming up…..
or maybe just showing a warning as mentioned above and you can click it and select ‚convert this‘ or ‚convert all‘….via a popup context menu….need to take a nap 😀

[john iv]

Yes these conversions take a while even on quick machines with beefy memory and cores.  I like the idea of being able to identify the candidates and then act on one or all of them; so folks could go one at a time to verify things are progressing properly and then turn the system loose on the rest overnight or whatever.  Like I said at Mameworld I've shaved off 200GB already on just the kpython2.cpp clone CHDs manually doing about 12 of them.

[Roman]

Yeah I think I will go with the popup context menu options….more tomorrow

[john iv]

One of the things that's taking time here manually doing it is that because I use split-sets the CHD clones are in their own directories so I have to copy the parent into the clone directory before kicking off the delta operation.  Are you accounting for this 'split' with your cmpro technique? Actually I wonder if chdman can take full paths for its -input -output arguments or if it expects them to be in the same folder.

[Roman]

Chdman takes full paths and I provide them whereever the clone and parent were found :-)
So….no need for manual copying/searching.

[Roman]

Coming closer:

- you can warn about non-deltas or deltas...or you don't care at all about deltas..
- convert or revert found single or multi (delta) chds after a scan (warnings appear in the warnings windows, popup menu gives you options for revert/convert)
- backup old abs/delta chds so nothing gets lost...(backup folder gets _chd_abs_ and _chd_delta_ subfolders)
- detects bad parent chds which are stored as delta (this is not allowed...you need to revert the file if possible)
- detects bad clone chds where the parent is 'lost', i.e. it was created with a different chd than the current parent (you need to revert the file if possible)
- revert possibility when 'parent' chd is found somewhere in your collection
- software lists included via part names

Generally, for converting the parent and clone file have to exist and they share the same "region"/"part name" in the datfile. For reverting, the chd which was used to generate the delta needs to be in your collection. The warnings appear after a scan in the warnings window.

doing some more testing and thinking about a good way when you only do a scan (not new scan) where not all chds are scanned….

by the way: delta converting a file and reverting it may result in a slightly different file….most likely based on the fact that the original chd file was created with a different chdman in the past with an older lzma or zlib version….the sha1 do match though…

[john iv]

Looks like good planning so far Roman.
[Update] Occurs to me that there may be potential with conversion and backing up original clone CHDs for folks to run out of HD space, does clrmame error out gracefully at that point?

[Roman]

If something fails during chdman conversion or you ctrl-c break out and are currently running multiple ones, you are prompted if you want to continue or not.
Generally, if something fails, the files are not touched...
So it's like

convert .chd -> .delta, if something fails, stop (or prompt in case of a being in a loop)
rename .chd -> .abs (if something fails, remove the .delta)
copy .abs to backup (if something fails, remove the .delta, rename .abs back to .chd)
rename .delta -> .chd (....you know what...)
if we came here, delete .abs

similar for reverting

hopefully I wrote that down without looking at the code ;-)