EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: [Feature Request] Heterogenous CHD  (Read 2995 times)

moviebird

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 4
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
[Feature Request] Heterogenous CHD
« on: 03 July 2020, 17:08 »

Would it be possible to add a feature to use a CHD as the archive wrapper for any item?

The use case I'm thinking of is No-Intro PSP dats. These are single iso's, which I'd like to compress to CHD. However I'd like to still be able to verify against the No-Intro dat.

I assume the issue has to do with the rom name having the file extension in it. But could we have a flag to ignore the extension if the sha1 matches?

Or would it be helpful if No-Intro separated the extension into another xml element, and leave the name as the strict name?

The extension of this would be to use on Redump dats. For instance, mednafen is a popular choice to run PlayStation CHDs.
Logged


Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
Re: [Feature Request] Heterogenous CHD
« Reply #1 on: 03 July 2020, 17:23 »

Sorry but I don't get it.

".chd" is the default extension for files which are listed as "disk" element in the datfile. Normally it's listed in the datfile without the extension.
Like <disk name="99bottles" sha1="0b874178c8dd3cfc451deb53dc7936dc4ad5a04f" region="cdrom" index="0" writable="no"/>

The given sha1 is not the sha1 of the file since a chd acts as a container (like  zip/7z/rar). The sha1 value is the internal, decompressed sha1 which is stored in the chd header for example.

I don't see any work here for cmpro....or I don't understand what you mean ;-)
Logged

moviebird

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 4
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
Re: [Feature Request] Heterogenous CHD
« Reply #2 on: 04 July 2020, 16:11 »

I'm not quite sure I understand "the given sha1 is not the sha1 of the file".

I think what you're telling me is that a MAME dat gives the hash for the container, i.e. the disk is compressed into a chd, and then the overall chd is hashed. This overall chd hash is what is given in the MAME dat.

But doesn't the chd store the uncompressed file hash in it's own header? So if I had 99bottles.chd, I could pull a hash from the header, which should match the uncompressed 99bottles?

So if we have any old file which is listed in an uncompressed state in a dat:

<rom name="Animal Crossing-e - Series 1 - Admiral (USA).raw" size="2912" crc="683bf051" md5="44339484138c4aa62bdf4e986780876e" sha1="661d50882467f2cce94f33528171dca3e8be31ca" status="verified"/>

Then doesn't your tool look for hte hash in the zip header when it scans Animal Crossing.zip? Which it then uses to compare against the dat?

If I have this right, then my thought is to have a flag to look inside the chd header for the uncompressed hash, and compare that against the dat entry. This would enable us to use standard dat files with any rom extension, but it's now compressed into a chd instead of a zip/7zip/rar/etc.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
Re: [Feature Request] Heterogenous CHD
« Reply #3 on: 04 July 2020, 17:16 »

A chd file is a compressed hunk of data, i.e. it compresses all kind of data (diskimages, dvdimages, floppyimages, laserdisk data, etc). It stores a chd header with all kind of information and the compressed data. See it like an archive like zip or rar.

The sha1 given in the datfile is the sha1 which chdman (the tool which can create chds/unpack chds, etc) calculates over the uncompressed data. This sha1 is stored in the chd header. Pretty much the same what a zip archive tool does. There it stores the CRC32 of the decompressed file in the zip structures.

So if it comes to any hash, it's always the hash of the decompressed data...and of course this hash is listed in the dat and this is the hash which clrmamepro checks against the file. Quick way is a look up in the structures, slower way is a blockwise decompress, calc the hash and compare it.

I still don't fully understand what your real idea is (most likely you're asking for a pretty obvious thing which chdman simply already does).
Again: "disk" elements refer to chdman-compressed data. There won't be any new flag which "to look inside the chd header". cmpro does that already which each chd file / disk entry. If your "any rom extension" data is compressable with chdman then nothing stops you.


Logged

moviebird

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 4
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
Re: [Feature Request] Heterogenous CHD
« Reply #4 on: 04 July 2020, 19:31 »

Let's say I have a zip file, game.zip, which holds game.iso. Let's now say that I take game.iso and convert it to game.chd.

Let's say it resides in directory /ROMs/game/game.chd

If I scan /ROMs/ with a No-Intro dat, it will tell me that game.chd is an unneeded file. However, /ROMs/game.zip scans as a valid file.

Really all I want is for ROMs/game.chd to scan as a valid file. Bonus points if the rebuilder gets an option to compress to chd.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
Re: [Feature Request] Heterogenous CHD
« Reply #5 on: 04 July 2020, 19:56 »

"Let's say I have a zip file, game.zip, which holds game.iso"

Ok..then you simply have a zipped up iso...the zip stores the compressed iso and also a crc32 for the file in its structures. So a valid datfile for this would hold a "rom" entry with the crc32 which can also be found when looking at the zip structures. For the sha1 value you can use common sha1 tools to calculate the sha1 from the iso file (not the zip).

"Let's now say that I take game.iso and convert it to game.chd."
Well. That means that you use chdman.exe!!! A simple rename of the extension is not what will work. A chd is a totally different format. In case you only renamed it from .iso to .chd you try to fool cmpro and reading it will fail since internally it will say that the chd file is not a valid file.

"If I scan /ROMs/ with a No-Intro dat, it will tell me that game.chd is an unneeded file"
Different cases could happen:
- the datfile doesn't hold "disk" elements...since it was designed for handling isos which are handled as a "rom"
- if you converted the file using chdman, then the datfile should list the sha1 which chdman reports when running an info command on it
- if you did not convert the file and just renamed it, well, then most likely cmpro tried to read it as chd, while it's not a chd and failed.


"Really all I want is for ROMs/game.chd to scan as a valid file."

Again...you need to follow the general rules: Either your datfile uses a "disk" element and the name/sha1 belong to a real chd (compressed with chdman) or you use a "rom" and the name/crc32/sha1 belong to any binary file

"Bonus points if the rebuilder gets an option to compress to chd."
No, since creating chds is the job for chdman.

I'm still not sure if you are aware of MAME's chdman tool which is reposible for packing data into a chd. It requires general information about the raw data you're trying to compress. Sectorsizes, etc..etc..

It's up to the datfile authors to define how the sets should look like. If nointro likes to store iso files 'as-is'. Then they're absolutely right to use a rom xml element and the hashes belong to that iso. Surely this iso can be zipped/rared/7z...but NOT put in a chd.
If nointro dat authors like to convert the isos into CHDs they will adjust the datfiles.

Again, I don't see anything which cmpro has to change here.


...besides of this...the datfile authors should be aware what the belonging emulators actually can handle. If emulators can only work with iso files it wouldn't make any sense to store them in chds and make dats for this....CHD is coming from the MAME universe...there might be a couple of emulators which can handle chds but globally it's still a rare format.
« Last Edit: 04 July 2020, 20:38 by Roman »
Logged

moviebird

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 4
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
Re: [Feature Request] Heterogenous CHD
« Reply #6 on: 05 July 2020, 14:58 »

Yes, game.iso is compressed to game.chd by way of chdman.exe. I think we've come full circle here. The hash in the header of game.chd is the hash of game.iso.

I think we're talking past each other on the general rules, however.

You state that either the dat file uses a disk element to use a chd, or a rom element for any binary file.

My request is to allow the use of chd as a container, i.e. not having it hardcoded to a disk tag. Or adding a flag so the user can mark the chd box in Hash & CHD, instead of having it greyed out. If I understand the usage of a chd correctly, it could be used as a container analogous to zip, by looking in the header for the hash. Obviously the creation of a chd is driven by chdman, so I wouldn't put a txt file inside.

I understand that MAME uses it specifically for a disk element, but more emulators are using it for general optical media storage.

The rebuilder comment was really more for allowing the Compress files option to list chd, which would follow the path listed for chdman in Compressor options.

Edit for some links:

Here's a post talking about adding CHD support:

https://www.libretro.com/index.php/category/chd/

and a library to readchds:

https://github.com/rtissera/libchdr
« Last Edit: 05 July 2020, 15:05 by moviebird »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 83.0.4103.116 Chrome 83.0.4103.116
    • View Profile
Re: [Feature Request] Heterogenous CHD
« Reply #7 on: 05 July 2020, 17:29 »

Yeah I think we're talking a bit in circles. But I can't stop showing the important difference of a "disk" and a "rom"

A "disk" element is used explicity for chds (so a compressed container holding specific kind of binary data where hashes are for the uncompressed data in. Actually you also have hashed for meta information).
A "rom" element is for any kind of binary data as is (i.e. not a container and the hashes are for the file itself).
All roms of a set are usually compressed together in an archive (which then represents the set) while chds are kept outside of the archive. Such rules aren't made by clrmame...it's all based in the MAME world.

You can't simply use chdman within the rebuilder or something to compress files like you do with a zip.
Different media (CD/DVD/HD/ect) require very specific and different additional parameters. Take a look at its commandline options for createld or createhd....there you need to specify sector sizes, inputstartframes etc etc.
The rebuilder rebuilds roms only. It informs the user about valid chds though but in the public build ( ;-)) it won't move the chd from source to destination

To make a long story short:

If you want to use chds for isos, fine, then the datfile should use disk elements.

This behaviour will not change.
chd -> disk xml element, no flag which changes this, no nothing. Sorry.
Logged
Pages: [1]   Go Up
 

Page created in 0.122 seconds with 20 queries.

anything
anything