EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: Delta CHDs new feature.  (Read 6764 times)

john iv

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 35
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Delta CHDs new feature.
« on: 21 October 2023, 17:55 »

Hi Roman, I assume you've seen:
https://github.com/mamedev/mame/commit/d1172bf710f2a7b1777ed7687f5ba65748d90ee9

This could lead to significant savings on child CHDs becoming deltas of their parents. Have you got your mind around how to support it? I'm not sure how to produce one of these deltas, but a guide might be nice for the potential space savings.
Logged


Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #1 on: 21 October 2023, 20:36 »

Well, I think regarding auditing nothing much will change. I haven't looked at the changes yet, but I assume that

a) the hash / name values of the disk entries in -listxml output doesn't change
b) the chd header will get some additional flag that shows that it is a delta chd

cmpro reads only the chd header and compares the hashes against the hashes in the xml output ...so that will still work....a deep check is done via chdman anywhow...so I think that will also still work since they probably update chdman's test routine...

....so in the end it might be only something like "couldn't cmpro show me that I am able to use a delta chd instead of a full one" request....

but hey...I might be wrong...time will tell....thanks for the headup
Logged

john iv

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 35
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #2 on: 21 October 2023, 20:41 »

I could see people abandoning the child .chds and just keeping the deltas, so was curious if you'd have to account for that.  Guess we'll see! :)

[Update] I just converted gtfore04a and it went from 1GB to a delta of 90MB.  Clrmame scan flagged the chd 'missing' though it was in the right place w/ the right name, fwiw.

[Update2] Roman, it would be cool if you could use Clrmame or one of your external tools to find eligable CHDs and prompt to create their deltas all in one pass.

chdman copy -i sf2049tea.chd -op sf2049te.chd -o sf2049tea-delta.chd
mv sf2049tea.chd sf2049tea-abs.chd
mv sf2049tea-delta.chd sf2049tea.chd
« Last Edit: 22 October 2023, 03:00 by john iv »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #3 on: 22 October 2023, 07:22 »

Yeah yeah keeping an old man busy....

Thanks for the testing and information....First thing I will do is of course checking why a delta is not recognized when it got the correct name in the correct place...most likely because the chd header changed a bit...

"fixing" or better "auto-create a delta" out of a found non-delta clone....hmm....will see if this is easy to add (well it is...but I have very limited time).

Question is if (or if it makes sense that) each and every clone chd can be converted to a delta compared to its parent. Haven't checked yet but I'm sure that there are sets with multiple chds and maybe a clone of it also has multiple ones...question is how to find the correct parent chd to the clone chd in such a multi-case.   ....most likely if the region entries match...like "region="pci:05.0:ide:0:hdd"" ...


to sum it up: I try my best to update cmpro as quick as possible....
« Last Edit: 22 October 2023, 08:25 by Roman »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #4 on: 22 October 2023, 12:26 »

ok...attached a quick version so that cmpro works again....can anyone give it a try for delta chds?

Thinking about

a) an optional warning if found clone disks could get replaced by a delta
b) maybe a standalone tool for converting....(guess that can be achieved quicker than adding some conversion thing into old cmpro..)

but don't count on a) and b) in the very near future....
Logged

john iv

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 35
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #5 on: 22 October 2023, 16:18 »

Roman, the new .exe worked.  It didn't flag the four CHDs I've converted.  Good job. :)
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #6 on: 23 October 2023, 18:07 »

ok..let's see what I got....and what needs to be done...

Already added the ability to:


- show an error entry (scanner, tree view) if a parent chd is a delta chd (parent chds can't be deltas, this can happen when MAME changed parent/clone relationships)
- show an error entry (scanner, tree view) if a clone chd is a delta chd but the parent sha1 stored in it doesn't match the current parent chd (as above, maybe some parent/clone relationships changed)

- optionally show a warning about a clone chd which can be a delta but isn't (warning window below scanner) (default: false)
- optionally auto converting a clone to a delta via chdman call (similar to do deep chd checks), see attached screenshot  (default: false)
- optionally doing a backup of the original clone chd before it gets converted to a delta  (default: false, gets auto-enabled with enabling converting option, but you can disable it afterwards)


To do:
- if the upper mentioned errors appear, you would need to revert the delta to the original image. For that, the right chd needs to be found first (could be outside of the current parent/clone relationship)
- software list chds are not yet supported. Currently a possible delta-conversion is detected if the parent/clone disks are available AND they share the same "region" information. In software lists you have "parts"...and I need to do some more investigation on them...if it's safe to do a match by using this information.

Generally, converting the chds takes a lot of time...plus the checks in cmpro are purely experimental for now.....Keep you updated.
« Last Edit: 23 October 2023, 18:27 by Roman »
Logged

john iv

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 35
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #7 on: 23 October 2023, 19:20 »

Looking good, see you have been interacting w/ Vas and getting good info on mameworld.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Mac OS X 10.15.7 Mac OS X 10.15.7
  • Browser:
  • Safari 17.0.1 Safari 17.0.1
    • View Profile
Re: Delta CHDs new feature.
« Reply #8 on: 23 October 2023, 19:46 »

yes….need to check how to revert a chd and I still think about the way a conversion makes sense….just having that option enabled can cause a long long scan but currently I dont have a good idea about a different way….I mean if users enable the option they know what is coming up…..
or maybe just showing a warning as mentioned above and you can click it and select ‚convert this‘ or ‚convert all‘….via a popup context menu….need to take a nap 😀
« Last Edit: 23 October 2023, 19:48 by Roman »
Logged

john iv

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 35
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #9 on: 23 October 2023, 20:02 »

Yes these conversions take a while even on quick machines with beefy memory and cores.  I like the idea of being able to identify the candidates and then act on one or all of them; so folks could go one at a time to verify things are progressing properly and then turn the system loose on the rest overnight or whatever.  Like I said at Mameworld I've shaved off 200GB already on just the kpython2.cpp clone CHDs manually doing about 12 of them.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Mac OS X Mac OS X
  • Browser:
  • Safari 17.0.1 Safari 17.0.1
    • View Profile
Re: Delta CHDs new feature.
« Reply #10 on: 23 October 2023, 20:22 »

Yeah I think I will go with the popup context menu options….more tomorrow
Logged

john iv

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 35
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #11 on: 23 October 2023, 23:20 »

One of the things that's taking time here manually doing it is that because I use split-sets the CHD clones are in their own directories so I have to copy the parent into the clone directory before kicking off the delta operation.  Are you accounting for this 'split' with your cmpro technique? Actually I wonder if chdman can take full paths for its -input -output arguments or if it expects them to be in the same folder.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Mac OS X Mac OS X
  • Browser:
  • Safari 17.0.1 Safari 17.0.1
    • View Profile
Re: Delta CHDs new feature.
« Reply #12 on: 24 October 2023, 04:33 »

Chdman takes full paths and I provide them whereever the clone and parent were found :-)
So….no need for manual copying/searching.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #13 on: 25 October 2023, 18:14 »

Coming closer:

- you can warn about non-deltas or deltas...or you don't care at all about deltas..
- convert or revert found single or multi (delta) chds after a scan (warnings appear in the warnings windows, popup menu gives you options for revert/convert)
- backup old abs/delta chds so nothing gets lost...(backup folder gets _chd_abs_ and _chd_delta_ subfolders)
- detects bad parent chds which are stored as delta (this is not allowed...you need to revert the file if possible)
- detects bad clone chds where the parent is 'lost', i.e. it was created with a different chd than the current parent (you need to revert the file if possible)
- revert possibility when 'parent' chd is found somewhere in your collection
- software lists included via part names

Generally, for converting the parent and clone file have to exist and they share the same "region"/"part name" in the datfile. For reverting, the chd which was used to generate the delta needs to be in your collection. The warnings appear after a scan in the warnings window.

doing some more testing and thinking about a good way when you only do a scan (not new scan) where not all chds are scanned….

by the way: delta converting a file and reverting it may result in a slightly different file….most likely based on the fact that the original chd file was created with a different chdman in the past with an older lzma or zlib version….the sha1 do match though…
« Last Edit: 25 October 2023, 20:02 by Roman »
Logged

john iv

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 35
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #14 on: 25 October 2023, 19:32 »

Looks like good planning so far Roman.
[Update] Occurs to me that there may be potential with conversion and backing up original clone CHDs for folks to run out of HD space, does clrmame error out gracefully at that point?
« Last Edit: 26 October 2023, 04:16 by john iv »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Online Online
  • Posts: 3324
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 118.0.0.0 Chrome 118.0.0.0
    • View Profile
Re: Delta CHDs new feature.
« Reply #15 on: 26 October 2023, 08:12 »

If something fails during chdman conversion or you ctrl-c break out and are currently running multiple ones, you are prompted if you want to continue or not.
Generally, if something fails, the files are not touched...
So it's like

convert .chd -> .delta, if something fails, stop (or prompt in case of a being in a loop)
rename .chd -> .abs (if something fails, remove the .delta)
copy .abs to backup (if something fails, remove the .delta, rename .abs back to .chd)
rename .delta -> .chd (....you know what...)
if we came here, delete .abs

similar for reverting

hopefully I wrote that down without looking at the code ;-)
« Last Edit: 26 October 2023, 08:12 by Roman »
Logged
Pages: [1]   Go Up
 

Page created in 0.192 seconds with 21 queries.