Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Roman]

Rebuilder 0.05 released

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.

As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).

RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.

Hope you enjoy it...

[AntoPISA]

Fantastic!
Thanks Roman 

[Pandor]

tried it, but how do I verify the output?
https://www.emulab.it/forum/index.php?topic=8905.msg25366#msg25366

[haynor666]

Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.

Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro

[Roman]

tried it, but how do I verify the output?
https://www.emulab.it/forum/index.php?topic=8905.msg25366#msg25366

The readme.md tells you:



Since there is no scanner yet, can I scan my rebuilt sets with clrmamepro?
Yes, but be aware of the following:

• clrmamepro does not have the standalone merge mode (it only has not-merged + disable separate bios sets, but either way it won't match)
• clrmamepro's profiler options to parse disk/rom merge tags and don't create dummy clones have to be enabled (that's the default by the way) since the new rebuilder only cares about merge attributes for checking relationships.
• for full merged sets, clrmamepro's settings option "Hash Collision Name" mode has to be enabled (this is not enabled by default) and profiler option "naming pattern" should be "%f%1" (default). The new rebuilder always uses subpaths for clones/dependencies and that matches this behaviour. Keep in mind, this is only valid for compressed sets. Decompressed ones don't have subfolders, then don't enable the hash collision name mode.

[Roman]

Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.

Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro

Generally the speed should be way faster than cmpro since the rebuilder is capable of copying sets directly (if they are ready for it). It may look slower on the first glance since scanning the source takes a bit while old cmpro took one file from the source and then rebuilt it when possible and so on....
Slowing down factors are: sha1 checks (without it, it flies, however you should know what you rebuild....crc32 collisions are not that uncommon), not setting a good temporary folders (in case unpack operations are necessary) and finally....solid 7z files sha1 enabled....this is a real crawler

But for example a sha1-less split zip-to-zip rebuild of let's say all NeoGeo sets is 18 seconds....fast enough I think......

[haynor666]

I disabled SHA1 for tests. rebuilder is put on M.2 disk as well togehter with mame.XML file. About 8 mins took scan of whole roms folder. About 26 mins. took 10% of rebuilding.

[Roman]

Don't forget, the new rebuilder rebuilds CHDS, too.....and multi gb chds can take a while to copy :-)  (ever tried it manually .... your system will be pretty busy)

I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace

[haynor666]

CHDs are removed for now

Target directory is on clean/formatted partition.

[Roman]

hmm...all mechanial ones (3 chds, ~15k sets), old cmpro (without chds): 210 seconds, new rebuilder (with chds): 109 seconds...
(split, no sha1, zip2zip, hd to (different) hd)

[haynor666]

Rebuilder reported 170398 unique roms. 23% and about 2 hours pass. I'm using rezip method but this probably does not matter since all roms are extracted from zip archives.

[Roman]

rezip...yikes...that of course unzips each file from each archive to temp and recompresses the files....why do you do that? If you want zip files in your destination, simply use "zip"....if something decompressed, something in a rar or 7z matches in the source it will be (unpacked) and repacked to zip....everything which is already zipped in the source can be copied without unpack/repacking....you're wasting time with rezip ;-)

[haynor666]

It's intentional - I'm not sure if source zip file is checked against any errors so I need everything rezipped. Anyway 46 % done and about 2 hours and 40 mins passed.

[Roman]

I pretty sure  a recursive commandline unzip -t on all archives afterwards would be faster ;-) but ok...

[Roman]

ok...looks like other people also give some feedback:

a complete 253 (roms + chds) split sets to "mode: full" rebuild, "compress: zip", "sha: none", on one and the same HD without deletion of source files: 3441,69 seconds....which is 57 minutes...

...but "use links: hard" was used....so...a little bit of cheating :-) but still...remarkable

[haynor666]

Whole process started exactly 16:00:00. 7:58 took scan rebuild process was ended exactly at 20:16:10.

Now let's see how it will take with use of clrmame.

[Roman]

Restart your machine first, otherwise you will cheat using the disk cache ;-)
...and of course you need to enable "recompress" in cmpro's rebuilder... (which corresponds to 'rezip')
Would be interesting to see....I mean with turning on "rezip" you of course disabled the major speed optimization....since it doesn't copy full archives in that case

[haynor666]

Rebuilded from the same source. Started at 7:15 ended about 12:35 so more than 5 hours was needed for entire process

[Roman]

Nice to hear...that old cmpro is slower ;-) I was a bit sceptical since you turned off all of the major speed advantages....but sounds good to me....

[haynor666]

Worth to mention that some zip archives have different crcs (for example 1941.zip) but size and files inside are exactly the same. Most of zips are exactly the same. I assume that clrmamepro adds some files into zips with different order.