EMULAB Forum
clrmamepro [English] => clrmame Discussion => Topic started by: Roman on 14 April 2023, 20:06
-
Rebuilder 0.05 released
https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html
0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.
As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).
RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.
Hope you enjoy it...
-
Fantastic!
Thanks Roman ;)
-
tried it, but how do I verify the output?
https://www.emulab.it/forum/index.php?topic=8905.msg25366#msg25366
-
Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.
Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro :)
-
tried it, but how do I verify the output?
https://www.emulab.it/forum/index.php?topic=8905.msg25366#msg25366
The readme.md tells you:
Since there is no scanner yet, can I scan my rebuilt sets with clrmamepro?
Yes, but be aware of the following:
- clrmamepro does not have the standalone merge mode (it only has not-merged + disable separate bios sets, but either way it won't match)
- clrmamepro's profiler options to parse disk/rom merge tags and don't create dummy clones have to be enabled (that's the default by the way) since the new rebuilder only cares about merge attributes for checking relationships.
- for full merged sets, clrmamepro's settings option "Hash Collision Name" mode has to be enabled (this is not enabled by default) and profiler option "naming pattern" should be "%f%1" (default). The new rebuilder always uses subpaths for clones/dependencies and that matches this behaviour. Keep in mind, this is only valid for compressed sets. Decompressed ones don't have subfolders, then don't enable the hash collision name mode.
-
Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.
Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro :)
Generally the speed should be way faster than cmpro since the rebuilder is capable of copying sets directly (if they are ready for it). It may look slower on the first glance since scanning the source takes a bit while old cmpro took one file from the source and then rebuilt it when possible and so on....
Slowing down factors are: sha1 checks (without it, it flies, however you should know what you rebuild....crc32 collisions are not that uncommon), not setting a good temporary folders (in case unpack operations are necessary) and finally....solid 7z files sha1 enabled....this is a real crawler
But for example a sha1-less split zip-to-zip rebuild of let's say all NeoGeo sets is 18 seconds....fast enough I think......
-
I disabled SHA1 for tests. rebuilder is put on M.2 disk as well togehter with mame.XML file. About 8 mins took scan of whole roms folder. About 26 mins. took 10% of rebuilding.
-
Don't forget, the new rebuilder rebuilds CHDS, too.....and multi gb chds can take a while to copy :-) (ever tried it manually .... your system will be pretty busy)
I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace
-
CHDs are removed for now ;)
Target directory is on clean/formatted partition.
-
hmm...all mechanial ones (3 chds, ~15k sets), old cmpro (without chds): 210 seconds, new rebuilder (with chds): 109 seconds...
(split, no sha1, zip2zip, hd to (different) hd)
-
Rebuilder reported 170398 unique roms. 23% and about 2 hours pass. I'm using rezip method but this probably does not matter since all roms are extracted from zip archives.
-
rezip...yikes...that of course unzips each file from each archive to temp and recompresses the files....why do you do that? If you want zip files in your destination, simply use "zip"....if something decompressed, something in a rar or 7z matches in the source it will be (unpacked) and repacked to zip....everything which is already zipped in the source can be copied without unpack/repacking....you're wasting time with rezip ;-)
-
It's intentional - I'm not sure if source zip file is checked against any errors so I need everything rezipped. Anyway 46 % done and about 2 hours and 40 mins passed.
-
I pretty sure a recursive commandline unzip -t on all archives afterwards would be faster ;-) but ok...
-
ok...looks like other people also give some feedback:
a complete 253 (roms + chds) split sets to "mode: full" rebuild, "compress: zip", "sha: none", on one and the same HD without deletion of source files: 3441,69 seconds....which is 57 minutes...
...but "use links: hard" was used....so...a little bit of cheating :-) but still...remarkable
-
Whole process started exactly 16:00:00. 7:58 took scan rebuild process was ended exactly at 20:16:10.
Now let's see how it will take with use of clrmame.
(https://i.ibb.co/9cwfLt0/rebuilder.png)
-
Restart your machine first, otherwise you will cheat using the disk cache ;-)
...and of course you need to enable "recompress" in cmpro's rebuilder... (which corresponds to 'rezip')
Would be interesting to see....I mean with turning on "rezip" you of course disabled the major speed optimization....since it doesn't copy full archives in that case
-
Rebuilded from the same source. Started at 7:15 ended about 12:35 so more than 5 hours was needed for entire process :)
-
Nice to hear...that old cmpro is slower ;-) I was a bit sceptical since you turned off all of the major speed advantages....but sounds good to me....
-
Worth to mention that some zip archives have different crcs (for example 1941.zip) but size and files inside are exactly the same. Most of zips are exactly the same. I assume that clrmamepro adds some files into zips with different order.
-
Hash values of an archive is of course obsolete.....order of added files and compression rate of course alters it....there is no guranteed add order, no sort, no nothing
-
I'm late this time but a huge THANK YOU for moving forward this! Will give it a spin ASAP
Rebuilder 0.05 released
https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html
0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.
As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).
RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.
Hope you enjoy it...
-
As you (the audience) might have noticed, "fool-for-a-day, troll-for-a-life-time" Jessica Jones, rolled over the boards again and was removed by admins....nothing unusual...this happens once a year...
But the really funny part is:
The discussion gave me an idea where threading can give some extra speed up....It's not related to the troll comments at all (CUDA for SHA1 etc...)
But a first small test was already pretty positive which gave a extra ~25% speed boost when sha1 input scanning was enabled....
For non sha1 runs it's not that big since you simply have pure lookups without decompression/sha1 and the actual gain by multiple threads is eaten up by the overhead to create the threads...
I will check how this scales for larger data and for number of threads....
So..Thank you Jessica Jones...you made my day. See you next year.
-
- parallelization? You don't want to have multiple threads treating your hd/ssd like crazy...ever tried to copy various bunch of files from one hd to another in multiple copy operations...it will crawl
But a first small test was already pretty positive which gave a extra ~25% speed boost when sha1 input scanning was enabled....
That's the way!! ;)
Thanks Roman 8)
-
There is a difference of parallel accessing the disk or doing parallel calculate threads….it needs a bit of tuning and checking to get a good result. If you force too many threads your cpu needs time for swapping etc… so as mentioned I will play around a bit. Dont expect too much….you also produce overhead with starting threads
-
doing parallel calculate threads….
Yup, I meant exactly this: bring computational (CPU) operations into parallel execution. 8)