241
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 16 April 2023, 15:11 »
no problem...such things do happen..
- 05 October 2024, 10:27
- Welcome, Guest
News:
The new forum is online, hope you enjoy it!
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
242
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 14:40 »
Don't forget, the new rebuilder rebuilds CHDS, too.....and multi gb chds can take a while to copy :-) (ever tried it manually .... your system will be pretty busy)
I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace
I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace
243
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 16 April 2023, 13:45 »Quote
As for CMP, (unless I'm doing something complete wrong here) it looks like even Rebuilder 0.05 can't produce a merged mame romset that can satisfy the scanner.. (no offence).Got several testers on board and noone reported a problem. So I guess you may have some setting wrongly set or you're using a wrong datasource (as reported before 239 vs 253)
Quote
how is it possible that alpinesa isn't listed as a clone to its parent alpines. in doing so, CMP wrongfully considers alpinesa as the parent and wants to
In general you can't fully merge alpinesa in alpines since they share an identically named rom with different hash (af2ver-a_ll.ic2). When cmpro imports data, it will prompt you about it (you most likely clicked on "ok to all" when the first of such events happend).
The hash collision detection will add a subfolder for it.... and it will show up as in the screenshot below.
Since you seem to have some weird effects which can not be repeated, it would be best if you send me your cmpro.ini, and the used .cmp file (cmpro's settings folder) and the used datfile (if any, if not, state which exe you're using).
244
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 16 April 2023, 13:36 »Quote
I understand that CMP, unlike RV might want to duplicate roms shared between clones if there is no merge TAG present. but not honoring the filesize and CRC as described in the XML seems weird.. Where is it getting this from?
Easy answer: You're tring to scan a 253 collection with a MAME binary based on 239 (the window title of your screenshot shows it). Back then the hashes were different.
245
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 13:32 »Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.
Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro
Generally the speed should be way faster than cmpro since the rebuilder is capable of copying sets directly (if they are ready for it). It may look slower on the first glance since scanning the source takes a bit while old cmpro took one file from the source and then rebuilt it when possible and so on....
Slowing down factors are: sha1 checks (without it, it flies, however you should know what you rebuild....crc32 collisions are not that uncommon), not setting a good temporary folders (in case unpack operations are necessary) and finally....solid 7z files sha1 enabled....this is a real crawler
But for example a sha1-less split zip-to-zip rebuild of let's say all NeoGeo sets is 18 seconds....fast enough I think......
246
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 13:29 »tried it, but how do I verify the output?
https://www.emulab.it/forum/index.php?topic=8905.msg25366#msg25366
The readme.md tells you:
Since there is no scanner yet, can I scan my rebuilt sets with clrmamepro?
Yes, but be aware of the following:
- clrmamepro does not have the standalone merge mode (it only has not-merged + disable separate bios sets, but either way it won't match)
- clrmamepro's profiler options to parse disk/rom merge tags and don't create dummy clones have to be enabled (that's the default by the way) since the new rebuilder only cares about merge attributes for checking relationships.
- for full merged sets, clrmamepro's settings option "Hash Collision Name" mode has to be enabled (this is not enabled by default) and profiler option "naming pattern" should be "%f%1" (default). The new rebuilder always uses subpaths for clones/dependencies and that matches this behaviour. Keep in mind, this is only valid for compressed sets. Decompressed ones don't have subfolders, then don't enable the hash collision name mode.
247
clrmame Discussion / rebuilder 0.05 released
« on: 14 April 2023, 20:06 »
Rebuilder 0.05 released
https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html
0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.
As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).
RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.
Hope you enjoy it...
https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html
0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.
As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).
RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.
Hope you enjoy it...
248
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 19:58 »
then you should switch to new rebuilder's standalone mode...which really produces "unmerged" sets ;-)
249
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 12:48 »
4TB HDs are < 100 EUR ......I come from a time where I paid > 800 (rough estimation) for an Amiga HD......for 32 MB.....
250
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 12:27 »
"As you say, I think it will deem it as unneeded, and move it to rev3 where it is 'needed', if not present, and discard it from rev4. But I have not tested this."
Why should it be needed in rev3 and not vice versa? just because some string says revision 3 versus 4 and an xml ordering of 3 before 4....Sorry...but I don't follow this logic.... :-)
You have parent/clone relationships of totally different kinds in MAME...where not even the setnames are in common...or the manufacturer etc....and yes, there you might also find identical files spread in the clones only...how do you decide there to which clone they really belong....
Again...if you think your/RomVault's full merge is more right, feel free to use it....there is no standard, there is no right way, it's just interpretation.
Why should it be needed in rev3 and not vice versa? just because some string says revision 3 versus 4 and an xml ordering of 3 before 4....Sorry...but I don't follow this logic.... :-)
You have parent/clone relationships of totally different kinds in MAME...where not even the setnames are in common...or the manufacturer etc....and yes, there you might also find identical files spread in the clones only...how do you decide there to which clone they really belong....
Again...if you think your/RomVault's full merge is more right, feel free to use it....there is no standard, there is no right way, it's just interpretation.
251
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 12:24 »
It's fun to hear the diskspace argumentation in 2023....if you decided to collect MAME (and maybe software lists) you shouldn't care about the few extra size of such "clones of clones" at all....each upcoming laserdisk chd will be more than all the files together...
I personally think "merged" or "split" was never a question of "hey I can save a few MB"...it's more about how you look at your collection. The "full merged" guys tend to "I want only one pacman archive and not pacman1 to pacman 20", while the "split merged" guys are more like "MAME support 1003 sets, I have 1003 archives and I can see the difference of pacman1 compared to pacman2...
"This way it does save (some marginal) space, doesn't it? as in my pit fighter example, rev3 becomes a parent for rev4" ....well, yes, but "becomes parent" is something romvault made up. There is no indicator in MAME's -listxml file for it. And yes, it saves marginal space
If you won't use subfolders for full merged sets, then you a) won't have the pitfighter roms doubled and b) you would even save more space since the folder name is not in the archive....
"What about split then? will CMP also include the rev3 roms into the rev4 zip?"
Split merged: you have 1 archive for each set, parent archives only hold the files for the parent, clone archives only hold files which are not marked as "merged". There is no sharing between clones
Full merged: you have only archives for parent sets, where the archive also holds the files for each clone.
To answer your question:
rev3 would hold: 136081-3028.05d, 136081-3029.05b, 136081-3030.15d, 136081-3031.15b (but not 136081-4028.05d or 136081-4029.05b)
rev4 would hold: 136081-4028.05d, 136081-4029.05b, 136081-3030.15d, 136081-3031.15b (but not 136081-3028.05d or 136081-3029.05b)
"merged is just the same as split, but clones are just included as subfolder in the parent/main rom, which could save some space because of zipfile header overhead"
the subfolders part is not really a part of merged sets. Originally there were no subfolders (and MAME does not need subfolders), subfolders were introduced first where it made sense (e.g. if you have clone files with identical name but a different hash)....but surely they give a better idea which file belongs to which clone if you always use them....but that latter argument also is an argument to have the clones-of-clones in each subfolder ;-)
I personally think "merged" or "split" was never a question of "hey I can save a few MB"...it's more about how you look at your collection. The "full merged" guys tend to "I want only one pacman archive and not pacman1 to pacman 20", while the "split merged" guys are more like "MAME support 1003 sets, I have 1003 archives and I can see the difference of pacman1 compared to pacman2...
"This way it does save (some marginal) space, doesn't it? as in my pit fighter example, rev3 becomes a parent for rev4" ....well, yes, but "becomes parent" is something romvault made up. There is no indicator in MAME's -listxml file for it. And yes, it saves marginal space
If you won't use subfolders for full merged sets, then you a) won't have the pitfighter roms doubled and b) you would even save more space since the folder name is not in the archive....
"What about split then? will CMP also include the rev3 roms into the rev4 zip?"
Split merged: you have 1 archive for each set, parent archives only hold the files for the parent, clone archives only hold files which are not marked as "merged". There is no sharing between clones
Full merged: you have only archives for parent sets, where the archive also holds the files for each clone.
To answer your question:
rev3 would hold: 136081-3028.05d, 136081-3029.05b, 136081-3030.15d, 136081-3031.15b (but not 136081-4028.05d or 136081-4029.05b)
rev4 would hold: 136081-4028.05d, 136081-4029.05b, 136081-3030.15d, 136081-3031.15b (but not 136081-3028.05d or 136081-3029.05b)
"merged is just the same as split, but clones are just included as subfolder in the parent/main rom, which could save some space because of zipfile header overhead"
the subfolders part is not really a part of merged sets. Originally there were no subfolders (and MAME does not need subfolders), subfolders were introduced first where it made sense (e.g. if you have clone files with identical name but a different hash)....but surely they give a better idea which file belongs to which clone if you always use them....but that latter argument also is an argument to have the clones-of-clones in each subfolder ;-)
252
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 11:51 »
would it list an additional 136081-3030.15d rom with f143f0e16850ad98366db208e956f7402d1ca848 in pitfight4 as unneeded then? Or would it simply mark it green?
Well, it's of course your decision, however if you're coming from MAME's -listxml output, you see that e.g. "136081-3030.15d" from pitfighter 3 and 4 do not have any "merge" attribute set...and if you follow the idea of having the archive structure like
- the archive is named after the parent set
- the subfolders in the archive are named after the clones and each holds the files for that clone but limits it to the ones which differ from its parent
Sounds a bit more structured if you ask me.....(compared to..."ah yeah...and it also needs a file from another subfolder since the clone shares a file with another clone, too)
But again...it's all about interpretation and personal taste.....however it's now 26 years that I hear of a full merged storing method like that (romvault's)...may be an idea as additional view for the rebuilder tool (https://www.emulab.it/forum/index.php?topic=8816.0) which currently supports split, full (always with subfolders) and "standalone" which is also "new" compared to cmpro....
Well, it's of course your decision, however if you're coming from MAME's -listxml output, you see that e.g. "136081-3030.15d" from pitfighter 3 and 4 do not have any "merge" attribute set...and if you follow the idea of having the archive structure like
- the archive is named after the parent set
- the subfolders in the archive are named after the clones and each holds the files for that clone but limits it to the ones which differ from its parent
Sounds a bit more structured if you ask me.....(compared to..."ah yeah...and it also needs a file from another subfolder since the clone shares a file with another clone, too)
But again...it's all about interpretation and personal taste.....however it's now 26 years that I hear of a full merged storing method like that (romvault's)...may be an idea as additional view for the rebuilder tool (https://www.emulab.it/forum/index.php?topic=8816.0) which currently supports split, full (always with subfolders) and "standalone" which is also "new" compared to cmpro....
253
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 11:19 »
First of all, it's all about interpretation. MAME itself doesn't care about filenames, it simply takes an archive and looks by hash for the needed file. So to make it playable, you don't need an audit tool....simply put all garbage into a file and you're done 
The interpretation for parent/clone relationships within cmpro is that clones can share a file with their parent but not with other clones.
The relationship is either based on identical names or on the "merge" atttribute in the datfile/-listxmloutput. By default, cmpro uses the merge attribute information (See profiler options "parse rom merge tags").
So yes...if you have Parent P with rom p1, Clone C1 with rom p1, c1 and Clone C2 with rom p1, c1, c2 and clone C3 with rom c3 only, you will end up in a merged set with:
p1
C1/c1
C2/c1 + C2/c2
C3/c3
That's the way it is, it won't change and hasn't changed since 1997 now..... :-)
There is no information that a version 3 is older or newer than a version 4 of a set. The relationship is defined in the datfile....if version is was designed to be the parent, version 4 and 3 could be clones...maybe a dev decides later on that 4 becomes parent....there is no real rule on that....one day it was "the most common version", then "the us or europe version", then the initial version .....there were lots of plans to set a rule on it...but actually I don't see one.
By the way, if they are identical (hash and name) you can get rid of the few extra bytes by simply NOT using subfolders for clones...then you only get them 1 time in the archive.....
The interpretation for parent/clone relationships within cmpro is that clones can share a file with their parent but not with other clones.
The relationship is either based on identical names or on the "merge" atttribute in the datfile/-listxmloutput. By default, cmpro uses the merge attribute information (See profiler options "parse rom merge tags").
So yes...if you have Parent P with rom p1, Clone C1 with rom p1, c1 and Clone C2 with rom p1, c1, c2 and clone C3 with rom c3 only, you will end up in a merged set with:
p1
C1/c1
C2/c1 + C2/c2
C3/c3
That's the way it is, it won't change and hasn't changed since 1997 now..... :-)
There is no information that a version 3 is older or newer than a version 4 of a set. The relationship is defined in the datfile....if version is was designed to be the parent, version 4 and 3 could be clones...maybe a dev decides later on that 4 becomes parent....there is no real rule on that....one day it was "the most common version", then "the us or europe version", then the initial version .....there were lots of plans to set a rule on it...but actually I don't see one.
By the way, if they are identical (hash and name) you can get rid of the few extra bytes by simply NOT using subfolders for clones...then you only get them 1 time in the archive.....
254
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 09:47 »
Hmm....nasty....try this: in Scanner, click on "split sets" and then again on "merged sets". Open SetInformation (bottom left button in scan results window), enter "m4pitfala" in prefered at top to jump to the pitfall set (remember your first screenshot...there you have the same.....), and click on a subitem like Pitfall (Empire) (MPU4, set 3)....on the right side you should see then "m4pitfalb\pf2_0.bin" listed (*with* subfolder).
Is this the case? If no, hmm....maybe you should remove the profile and create a new one...If yes, then please do a new scan and show one single error message again
Is this the case? If no, hmm....maybe you should remove the profile and create a new one...If yes, then please do a new scan and show one single error message again
255
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 09:36 »
Check: Profiler->Options-> Naming Pattern should be "%f\%1" (which is default by the way)
Ensure that Settings->Full Merge Mode -> "Hash Collision Name" is ticked, leave cmpro, start it, click on Profiler "Clear Cache" and reload the profile....
Ensure that Settings->Full Merge Mode -> "Hash Collision Name" is ticked, leave cmpro, start it, click on Profiler "Clear Cache" and reload the profile....
256
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 09:17 »
Looks like you're using full merged sets where all client files are *always* stored in subfolders.
By default, cmpro only does this when a merge collision was detected to prevent overwriting of files (when a clone uses an identically named rom file but with a different hash).
If you want to always use subfolders, go to Settings and turn on Full Merge Mode -> Hash Collision Name
By default, cmpro only does this when a merge collision was detected to prevent overwriting of files (when a clone uses an identically named rom file but with a different hash).
If you want to always use subfolders, go to Settings and turn on Full Merge Mode -> Hash Collision Name
257
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 13 April 2023, 08:49 »
Actually, you don't need a datfile, use a direct exe import....but if it's a clean -listxml redirected output, fine...
Profiler->Options->Parse ROM 'merge' tags and Parse DISK 'merge' tags should be enabled
And please give a (one) detailed set/rom example about what cmpro complains about....I'm pretty sure cmpro is right :-)
Profiler->Options->Parse ROM 'merge' tags and Parse DISK 'merge' tags should be enabled
And please give a (one) detailed set/rom example about what cmpro complains about....I'm pretty sure cmpro is right :-)
258
clrmame Discussion / Re: Rebuilder 0.04 released
« on: 04 April 2023, 12:11 »
- parallelization? You don't want to have multiple threads treating your hd/ssd like crazy...ever tried to copy various bunch of files from one hd to another in multiple copy operations...it will crawl
- the purpose of the rebuilder is to generate matched files, not sets. Besides of that, it can happen that you examine a million files, and the "final complete" sets can be generated of ouf the first five files you scan and the last three ones...so...no...not the purpose of a rebuilder
- cache? in which context?
- later...when one day the source is out, somebody will be able to generate a native linux executable from it...
- ziplibrary has the advantage of a supported copy operation which can transfer single parts of the archive without recompression.....and I own a full licence :-)
- the purpose of the rebuilder is to generate matched files, not sets. Besides of that, it can happen that you examine a million files, and the "final complete" sets can be generated of ouf the first five files you scan and the last three ones...so...no...not the purpose of a rebuilder
- cache? in which context?
- later...when one day the source is out, somebody will be able to generate a native linux executable from it...
- ziplibrary has the advantage of a supported copy operation which can transfer single parts of the archive without recompression.....and I own a full licence :-)
259
clrmame Discussion / Re: Rebuilder 0.04 released
« on: 03 April 2023, 11:54 »
Scanner will come (one day) and it will be commandline driven for a start.....but don't ask me when...time is limited as usual
260
clrmame Discussion / Re: Rebuilder 0.04 released
« on: 03 April 2023, 07:04 »
Well, yes, a scanner is planned (as well as some kind of profiler/UI) but I haven't started it yet...currently I do some polishing on the rebuilder core which should be done pretty soon.....