EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: clrmamepro generates duplicated files  (Read 5027 times)

elsanto

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 6
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 60.0 Firefox 60.0
    • View Profile
clrmamepro generates duplicated files
« on: 11 September 2018, 21:49 »

Hi all!

First of all, I wanted to thank you for such a great piece of software.

I have used clrmamepro with much success for several years, mainly for mame romset maintenance.

But now I found a problem. I want to rebuild a split romset for mame 191. I built mame and launched it with -listxml to generate a dat file. I loaded such file in profile window and set an empty folder as ROM path and some other folders with roms as ADD paths. Then i clicked on the rebuilder button and make sure that "Use Add-Paths" was checked, Destination folder was the empty folder and "Split Sets" was checked too. After rebuilding I found than the clrmamepro make many identical files in the ROM path. For example: kovlsjba.zip, kovlsjb.zip, kovlsqh.zip and kovqhsgs.zip. Those four files have all the same sha1sum (75e40d829b04b20028008d287bf11875458e77df). It's like if it was using some merged setting, despite I the checked "Split" option.

Is this normal? Am I missing something?. Can you help me, please?

I'm running cmp4035-32bits portable with wine under debian linux.

Thank you and regards.
Logged


Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 69.0.3497.92 Chrome 69.0.3497.92
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #1 on: 12 September 2018, 07:00 »

Nothing's wrong here.

First of all, there are sets which simply have identical clones, where e.g. only the internal driver handling differs or just a different name (which corresponds to the IC on the board name).

Secondly, keep in mind what the rebuilder is. It's not set based, it's file based, i.e. it only creates matching files (checksum compare against database) in the destination, not necessary full sets. If you now got for example 2 clones which are 80% identical and the rest 20% aren't, and you only rebuild files which belong to the identical ones, you end up with 2 incomplete clones which are fully identical. The mentioned sets share a lot of files, kovlsjba kovlsjb as clones only differ in 1 file (*prg.rom), maybe you don't have it, and so it rest of these 2 are fully identical.

You're looking at various clones by the way, not at a parent/clone. There is no merge definition between clones, so identical files within clones are pretty common, if they aren't shared with the parent.

And finally, there is a profiler->options setting which allows parsing "merge" information provided by MAME. By default, these settings are enabled (which is good). If you disable them, cmpro does a merge check by not looking at hashes only but also at names...so byte-identical files which don't share the same name are handled as 'different'.

But again, merging is something between a specified parent set and its clones, not clones against clones.
Logged

elsanto

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 6
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 60.0 Firefox 60.0
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #2 on: 12 September 2018, 18:29 »

OK, Roman. Thank you very much for the explanation.

I didn't know that some clones share roms between them but not with the parent. That explains the "kovlsjba/kovlsjb" case.

After reading your post, I realized that there is many shared files between romsets without a parent/clone relationship. Now, I'm rebuilding a merged set. I suppose that doing it this way, the collection will take much less space.

Maybe, the MAME team could develop a new "rom to game" assignations to store the sets more efficiently, as they did, many years ago, when they introduced the merged or split romset, to make collections smaller.

What I still don't understand is why are there is "identical clones". Would not be a smarter choice if two (or more) drivers could use the same romset instead of require "identical clones"?

Thank you again and regards.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Mac OS X Mac OS X
  • Browser:
  • Safari 10.0 Safari 10.0
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #3 on: 12 September 2018, 19:09 »

in times of cheap multi terabyte hds space isn't a real problem...

as an example for identical sets (however it is a parent/clone, not a clone/clone), there is natodef and natodefa, which only differ in the order of levels...which is defined by how mame loads the roms....both machines do exist in reality....and so you have 2 identical sets :-)

I bet there are others where rom sets are identical but chds images differ....or bios rom usage...all these are examples for ending with identical rom sets.
Logged

elsanto

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 6
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 60.0 Firefox 60.0
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #4 on: 13 September 2018, 20:47 »

as an example for identical sets (however it is a parent/clone, not a clone/clone), there is natodef and natodefa, which only differ in the order of levels...which is defined by how mame loads the roms....both machines do exist in reality....and so you have 2 identical sets :-)

Amazing!

Finally, I decided to store the roms collection as a merged set. The space saving is small, about 1,5% but I found that clrmamepro is much faster when rebuilding, scanning, etc... with merged sets.

Thank you and regards!
Logged

elsanto

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 6
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 60.0 Firefox 60.0
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #5 on: 14 September 2018, 23:07 »

as an example for identical sets (however it is a parent/clone, not a clone/clone), there is natodef and natodefa, which only differ in the order of levels...which is defined by how mame loads the roms....both machines do exist in reality....and so you have 2 identical sets :-)

I think I found another case: jdredd and jdreddb (at least with 0.78 DAT). But, in this case, the Scanner shows 3 warnings (on for each file inside the rom) regarding to jdreddb: "wrong placed file" and suggest to move to parents. With natodef and natodefa it doesn't show any error/warning.

I attached the relevant parts of dat file. I don't understand why the different behavior. Maybe is because jdredd/jdreddb requieres a CHD?

Thank you and regards.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 69.0.3497.92 Chrome 69.0.3497.92
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #6 on: 15 September 2018, 12:58 »

if you switched to full merged sets, then of course the judge dredd clones need to be put to the parent.

Hmm...and if you have natodefa and natodef present and full merge mode enabled, it should also list the natodefa files as wrong placed or wrong merged...
« Last Edit: 15 September 2018, 14:01 by Roman »
Logged

elsanto

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 6
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 60.0 Firefox 60.0
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #7 on: 16 September 2018, 20:11 »

Ooops, sorry. I was talking about two different sets.

The first set, the one I was talking about in the first three messages, was a "repository" romset that i decided to store fully merged.

In my last message I was talking about another set I have for recalbox (MAME 0.106 DAT). This one is using splitted roms and is the one who have the problem with jdredd and jdreddb.

I've noticed that ClrMamePro doesn't show any warning if there is a empty folder called jdreddb.

Again, sorry for the confussion, my bad... O:-)

Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 69.0.3497.92 Chrome 69.0.3497.92
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #8 on: 17 September 2018, 05:40 »

hmm..empty folder....hmm..ok...it should complain about it (that it should be merged to its parent)...but yes, could be that some "set-got-chd" check hides it....I will double check it. Thanks
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 69.0.3497.92 Chrome 69.0.3497.92
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #9 on: 17 September 2018, 16:06 »

hmm....I made a test and an empty (actually this doesn't play a role) jdreddb folder would lead to a message that it should be merged to its parent jdredd when using full merged mode....
so...maybe you can specify or minimize the problem a bit...this was tested with latest official MAME though....
Logged

elsanto

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 6
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 60.0 Firefox 60.0
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #10 on: 17 September 2018, 19:54 »

Ok. I will try to explain myself a bit better.

I have a MAME 0.106 dat file that I use to prepare games for advmame/raspberrypi.

I did a rebuild using as Add Paths some folders that I have with roms. I chosed split roms.

The problem is that clrmamepro did not create a jdreddb.zip file because the 3 roms that should be inside that zip file were already in jdredd.zip file.

If I run the scanner over the just rebuilded collection, clrmamepro says that jdreddb is a missing romset. To get a "no missing rom" result from clrmamepro scanner i can either:
- create a empty folder named jdreddb in rom path or....
- ... make a copy of jdredd.zip named jdreddb.zip (In this case the scanner shows a warning telling that the 3 roms inside jdreddb should be move to parent)

It's not a big problem for me. I just described it trying to find some differences between this case and natodef one. With natodef this problem does not happen.

Thank you and regards.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 69.0.3497.100 Chrome 69.0.3497.100
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #11 on: 18 September 2018, 06:49 »

if I remember correctly jdreddb differs from its parent only in the used chd file. So if you don't have that clone chd file present somewhere, it is possible that the set is marked as missing.

But I will try to repeat your detailed described way later at home...
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 69.0.3497.100 Chrome 69.0.3497.100
    • View Profile
Re: clrmamepro generates duplicated files
« Reply #12 on: 19 September 2018, 15:48 »

well, I was right...
the clone set jdreddb only consists of a chd, the clone does not have any roms on its own. So as long as you don't have the chd, you don't have the set and cmpro complains about a missing set and if you have the chd check enabled, it will list jdreddb as missing set and missing chd jdreddb.chd.

So everything's fine...you don't have the clone set since you don't have the chd (and that's the only part of the jdreddb clone which exists in a split scenario).

However you can fool cmpro by adding an empty folder jdreddb as container for the chd and the missing set message will go away.
Logged
Pages: [1]   Go Up
 

Page created in 0.157 seconds with 19 queries.