Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Pandor]

Hi,

I'm trying to verify a MAME 0.253 merged romset, But Clrmamepro 4.047 is reporting a whole lot of missing sets and wrong filenames. DAT/xml was created from mame.exe. default Clrmamepro settings. only change made is Merged sets selected in scanner window.

Romvault reports a 100% valid set with the same mame.exe xml output, so i kow for a fact the set is good.

I found this old post, with similar report, here.

Anyone know if there is any (default)setting that could cause this?

[Roman]

Actually, you don't need a datfile, use a direct exe import....but if it's a clean -listxml redirected output, fine...

Profiler->Options->Parse ROM 'merge' tags and Parse DISK 'merge' tags should be enabled

And please give a (one) detailed set/rom example about what cmpro complains about....I'm pretty sure cmpro is right :-)

[Pandor]

For romvault I extraxted the XML manually. For CMP, it is a direct import from mame.exe.
Merge tag options are 'on' as per default settings for Clrmamepro 4.047 (this is a new freshly unzipped, first usage case).

steps to reproduce:

• extract and run Clrmamepro 4.047
• create profile from mame.exe and load
• add rompath in settings
• set scanner option to prefer merged sets
• click new scan

as per request, screenshot of the scan, and proof the rom actually exist, despite CMP claiming the rom is not there (and/or wrong filename):

[Roman]

Looks like you're using full merged sets where all client files are *always* stored in subfolders.
By default, cmpro only does this when a merge collision was detected to prevent overwriting of files (when a clone uses an identically named rom file but with a different hash).
If you want to always use subfolders, go to Settings and turn on Full Merge Mode -> Hash Collision Name

[Pandor]

Looks like you're using full merged sets where all client files are *always* stored in subfolders.
By default, cmpro only does this when a merge collision was detected to prevent overwriting of files (when a clone uses an identically named rom file but with a different hash).
If you want to always use subfolders, go to Settings and turn on Full Merge Mode -> Hash Collision Name

Thanks for the detailed explanation.
I believe subfolders are the default for romvault, hence it verifies as 100%.

Prior to trying CMP, I used romvault to go from an older split set, to the latest merged romset, i'm trying to validate with CMP.
My set also verifies with the (torrent)source of the current set.

unfortunately, setting Hash Collision Name in settings and performing a new scan did not fix it for me.

[Roman]

Check: Profiler->Options-> Naming Pattern should be "%f\%1" (which is default by the way)

Ensure that Settings->Full Merge Mode -> "Hash Collision Name"  is ticked, leave cmpro, start it, click on Profiler "Clear Cache" and reload the profile....

[Pandor]

followed your suggestions, but still same result.

[Roman]

Hmm....nasty....try this: in Scanner, click on "split sets" and then again on "merged sets". Open SetInformation (bottom left button in scan results window), enter "m4pitfala" in prefered at top to jump to the pitfall set (remember your first screenshot...there you have the same.....), and click on a subitem like Pitfall (Empire) (MPU4, set 3)....on the right side you should see then "m4pitfalb\pf2_0.bin" listed (*with* subfolder).

Is this the case? If no, hmm....maybe you should remove the profile and create a new one...If yes, then please do a new scan and show one single error message again

[Pandor]

CMP does (now) show the subfolder in front of the rom in the SetInformation, and pitfall seems okay now.

But now i'm left with a large list of "missing but fixable"

Seems like pit fighter rev4 want the include roms that are from pit fighter rev3 (and are present).
I don't believe this should be by design for a merged romset, as the proceeding revision already has the correct files.

settings and profile are correct (I believe):


Ater all this,
I tried starting over from a freshly unzipped CMP, reloading mame.exe,.... nothing seems to do the trick.

[Roman]

First of all, it's all about interpretation. MAME itself doesn't care about filenames, it simply takes an archive and looks by hash for the needed file. So to make it playable, you don't need an audit tool....simply put all garbage into a file and you're done

The interpretation for parent/clone relationships within cmpro is that clones can share a file with their parent but not with other clones.
The relationship is either based on identical names or on the "merge" atttribute in the datfile/-listxmloutput. By default, cmpro uses the merge attribute information (See profiler options "parse rom merge tags").

So yes...if you have Parent P with rom p1, Clone C1 with rom p1, c1 and Clone C2 with rom p1, c1, c2 and clone C3 with rom c3 only, you will end up in a merged set with:

p1
C1/c1
C2/c1 + C2/c2
C3/c3

That's the way it is, it won't change and hasn't changed since 1997 now..... :-)

There is no information that a version 3 is older or newer than a version 4 of a set. The relationship is defined in the datfile....if version is was designed to be the parent, version 4 and 3 could be clones...maybe a dev decides later on that 4 becomes parent....there is no real rule on that....one day it was "the most common version", then "the us or europe version", then the initial version .....there were lots of plans to set a rule on it...but actually I don't see one.


By the way, if they are identical (hash and name) you can get rid of the few extra bytes by simply NOT using subfolders for clones...then you only get them 1 time in the archive.....

[Pandor]

Thanks. So as you say, and what I suspected, it is subject to interpretation (there is no standard unfortunately, and MAME does indeed not care, as long as the file is there somewhere), and it seems CMP and RV differ in that aspect. so a RV merged romset, is not compatible with CMP's algorithm, apparently (and vice/versa).

I've always been a split set advocate, but thought I would change that for this current set. I guess it bit me in the *ss 

I'll just stick to RomVault then. it is, in my humble personal opinion a bit more intuitive and user-friendly.

Thanks for the help though. It did clear up a lot.

[Pandor]

Just for reference, this is how RomVault interprets the same set:

[Roman]

would it list an additional 136081-3030.15d rom with f143f0e16850ad98366db208e956f7402d1ca848 in pitfight4 as unneeded then? Or would it simply mark it green?



Well, it's of course your decision, however if you're coming from MAME's -listxml output, you see that e.g. "136081-3030.15d" from pitfighter 3 and 4 do not have any "merge" attribute set...and if you follow the idea of having the archive structure like

- the archive is named after the parent set
- the subfolders in the archive are named after the clones and each holds the files for that clone but limits it to the ones which differ from its parent

Sounds a bit more structured if you ask me.....(compared to..."ah yeah...and it also needs a file from another subfolder since the clone shares a file with another clone, too)

But again...it's all about interpretation and personal taste.....however it's now 26 years that I hear of a full merged storing method like that (romvault's)...may be an idea as additional view for the rebuilder tool (https://www.emulab.it/forum/index.php?topic=8816.0) which currently supports split, full (always with subfolders) and "standalone" which is also "new" compared to cmpro....

[Pandor]

would it list an additional 136081-3030.15d rom with f143f0e16850ad98366db208e956f7402d1ca848 in pitfight4 as unneeded then? Or would it simply mark it green?



Well, it's of course your decision, however if you're coming from MAME's -listxml output, you see that e.g. "136081-3030.15d" from pitfighter 3 and 4 do not have any "merge" attribute set...and if you follow the idea of having the archive structure like

- the archive is named after the parent set
- the subfolders in the archive are named after the clones and each holds the files for that clone but limits it to the ones which differ from its parent

Sounds a bit more structured if you ask me.....(compared to..."ah yeah...and it also needs a file from another subfolder since the clone shares a file with another clone, too)

But again...it's all about interpretation and personal taste.....however it's now 26 years that I hear of a full merged storing method like that (romvault's)...may be an idea as additional view for the rebuilder tool (https://www.emulab.it/forum/index.php?topic=8816.0) which currently supports split, full (always with subfolders) and "standalone" which is also "new" compared to cmpro....

This way it does save (some marginal) space, doesn't it? as in my pit fighter example, rev3 becomes a parent for rev4, and rev4 only includes roms that are not in rev3. in CMP's case, the roms from rev3 are duplicated from rev3 to rev4 and only the actual (head) parent, pitfight is taken into account, right?
I'm not saying this is the 'correct' way. Just thinking out loud, and trying to figure out the difference between CMP en RV, and why they differ.

What about split then? will CMP also include the rev3 roms into the rev4 zip?

Becuase the way I see it now (I could be completely wrong), merged is just the same as split, but clones are just included as subfolder in the parent/main zip, which could save some space because of zipfile header/encapsulation overhead?

[Pandor]

would it list an additional 136081-3030.15d rom with f143f0e16850ad98366db208e956f7402d1ca848 in pitfight4 as unneeded then? Or would it simply mark it green?
...

re-reading this, I get your point, and is something i'll test. Just like mame, it is possible that RV doesn't care where the file/rom is. and it will mark it green eighter way. but it is still a rom manager, so there should be some 'standard' it adheres to, and not put it randomly in a subfolder/zip that is most convenient, or it would never be consistent.

As you say, I think it will deem it as unneeded, and move it to rev3 where it is 'needed', if not present, and discard it from rev4. But I have not tested this.

[Roman]

It's fun to hear the diskspace argumentation in 2023....if you decided to collect MAME (and maybe software lists) you shouldn't care about the few extra size of such "clones of clones" at all....each upcoming laserdisk chd will be more than all the files together...

I personally think "merged" or "split" was never a question of "hey I can save a few MB"...it's more about how you look at your collection. The "full merged" guys tend to "I want only one pacman archive and not pacman1 to pacman 20", while the "split merged" guys are more like "MAME support 1003 sets, I have 1003 archives and I can see the difference of pacman1 compared to pacman2...

"This way it does save (some marginal) space, doesn't it? as in my pit fighter example, rev3 becomes a parent for rev4"  ....well, yes, but "becomes parent" is something romvault made up. There is no indicator in MAME's -listxml file for it. And yes, it saves marginal space

If you won't use subfolders for full merged sets, then you a) won't have the pitfighter roms doubled and b) you would even save more space since the folder name is not in the archive....

"What about split then? will CMP also include the rev3 roms into the rev4 zip?"

Split merged: you have 1 archive for each set, parent archives only hold the files for the parent, clone archives only hold files which are not marked as "merged". There is no sharing between clones
Full merged: you have only archives for parent sets, where the archive also holds the files for each clone.

To answer your question:
rev3 would hold: 136081-3028.05d, 136081-3029.05b, 136081-3030.15d, 136081-3031.15b (but not 136081-4028.05d or 136081-4029.05b)
rev4 would hold: 136081-4028.05d, 136081-4029.05b, 136081-3030.15d, 136081-3031.15b (but not 136081-3028.05d or 136081-3029.05b)


"merged is just the same as split, but clones are just included as subfolder in the parent/main rom, which could save some space because of zipfile header overhead"

the subfolders part is not really a part of merged sets. Originally there were no subfolders (and MAME does not need subfolders), subfolders were introduced first where it made sense (e.g. if you have clone files with identical name but a different hash)....but surely they give a better idea which file belongs to which clone if you always use them....but that latter argument also is an argument to have the clones-of-clones in each subfolder ;-)

[Roman]

"As you say, I think it will deem it as unneeded, and move it to rev3 where it is 'needed', if not present, and discard it from rev4. But I have not tested this."

Why should it be needed in rev3 and not vice versa? just because some string says revision 3 versus 4 and an xml ordering of 3 before 4....Sorry...but I don't follow this logic.... :-)

You have parent/clone relationships of totally different kinds in MAME...where not even the setnames are in common...or the manufacturer etc....and yes, there you might also find identical files spread in the clones only...how do you decide there to which clone they really belong....

Again...if you think your/RomVault's full merge is more right, feel free to use it....there is no standard, there is no right way, it's just interpretation.

[Pandor]

It's fun to hear the diskspace argumentation in 2023....if you decided to collect MAME (and maybe software lists) you shouldn't care about the few extra size of such "clones of clones" at all....each upcoming laserdisk chd will be more than all the files together...
...

I come from a time where a MAME collection would fit on a CD-rom... 
I had a spare 1TB drive, devoted to MAME (started years back), up till now. The past weekend I decided to fire it up again, and update my (then) split sets, but came to the conclusion the drive wasn't going to cut it anymore. so I decided to try my luck with a merged romset.

The (popular*)source I got it from, seems to use romvault as a means to manage the roms. So this is why I got intrigued into the reason why it does not match a CMP scan. Your replies have been verry educational and helpful, for which my gratitude.

For me it actually doesn't matter if I have a set as a single zip, or 20 zipfiles, but storage has become a 'problem', as in "i'll need to get myself a bigger disk for MAME now".

[Roman]

4TB HDs are < 100 EUR ......I come from a time where I paid > 800 (rough estimation) for an Amiga HD......for 32 MB.....

[Pandor]

"As you say, I think it will deem it as unneeded, and move it to rev3 where it is 'needed', if not present, and discard it from rev4. But I have not tested this."

Why should it be needed in rev3 and not vice versa? just because some string says revision 3 versus 4 and an xml ordering of 3 before 4....Sorry...but I don't follow this logic.... :-)

You are absolutely right. Maybe it's just my human logic, that a previous revision (or named so), would be a 'parent' to a latter revision, and that a child could receive from its parent... Don't get me wrong. you make a lot of sense. I'm not trying to undermine, just understand.