EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1] 2   Go Down

Author Topic: rebuilder 0.05 released  (Read 6554 times)

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
rebuilder 0.05 released
« on: 14 April 2023, 20:06 »

Rebuilder 0.05 released

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.

As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).

RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.

Hope you enjoy it...

« Last Edit: 14 April 2023, 20:22 by Roman »
Logged


AntoPISA

  • EMULAB Friend
  • Member
  • ***
  • Karma: 0
  • Offline Offline
  • Posts: 127
  • Operating System:
  • Linux Linux
  • Browser:
  • Chrome 106.0.5249.126 Chrome 106.0.5249.126
    • View Profile
    • progetto-SNAPS.net
Re: rebuilder 0.05 released
« Reply #1 on: 15 April 2023, 05:33 »

Fantastic!
Thanks Roman  ;)
Logged

Pandor

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 15
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #2 on: 15 April 2023, 14:51 »

Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 111.0.0.0 Chrome 111.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #3 on: 16 April 2023, 13:21 »

Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.

Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro :)
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #4 on: 16 April 2023, 13:29 »

tried it, but how do I verify the output?
https://www.emulab.it/forum/index.php?topic=8905.msg25366#msg25366


The readme.md tells you:



Since there is no scanner yet, can I scan my rebuilt sets with clrmamepro?
Yes, but be aware of the following:

  • clrmamepro does not have the standalone merge mode (it only has not-merged + disable separate bios sets, but either way it won't match)
  • clrmamepro's profiler options to parse disk/rom merge tags and don't create dummy clones have to be enabled (that's the default by the way) since the new rebuilder only cares about merge attributes for checking relationships.
  • for full merged sets, clrmamepro's settings option "Hash Collision Name" mode has to be enabled (this is not enabled by default) and profiler option "naming pattern" should be "%f%1" (default). The new rebuilder always uses subpaths for clones/dependencies and that matches this behaviour. Keep in mind, this is only valid for compressed sets. Decompressed ones don't have subfolders, then don't enable the hash collision name mode.

« Last Edit: 16 April 2023, 14:21 by Roman »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #5 on: 16 April 2023, 13:32 »

Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.

Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro :)


Generally the speed should be way faster than cmpro since the rebuilder is capable of copying sets directly (if they are ready for it). It may look slower on the first glance since scanning the source takes a bit while old cmpro took one file from the source and then rebuilt it when possible and so on....
Slowing down factors are: sha1 checks (without it, it flies, however you should know what you rebuild....crc32 collisions are not that uncommon), not setting a good temporary folders (in case unpack operations are necessary) and finally....solid 7z files sha1 enabled....this is a real crawler

But for example a sha1-less split zip-to-zip rebuild of let's say all NeoGeo sets is 18 seconds....fast enough I think......

« Last Edit: 16 April 2023, 13:50 by Roman »
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #6 on: 16 April 2023, 14:35 »

I disabled SHA1 for tests. rebuilder is put on M.2 disk as well togehter with mame.XML file. About 8 mins took scan of whole roms folder. About 26 mins. took 10% of rebuilding.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #7 on: 16 April 2023, 14:40 »

Don't forget, the new rebuilder rebuilds CHDS, too.....and multi gb chds can take a while to copy :-)  (ever tried it manually .... your system will be pretty busy)

I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace
« Last Edit: 16 April 2023, 14:41 by Roman »
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #8 on: 16 April 2023, 14:43 »

CHDs are removed for now ;)

Target directory is on clean/formatted partition.
« Last Edit: 16 April 2023, 14:44 by haynor666 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #9 on: 16 April 2023, 15:38 »

hmm...all mechanial ones (3 chds, ~15k sets), old cmpro (without chds): 210 seconds, new rebuilder (with chds): 109 seconds...
(split, no sha1, zip2zip, hd to (different) hd)
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #10 on: 16 April 2023, 16:15 »

Rebuilder reported 170398 unique roms. 23% and about 2 hours pass. I'm using rezip method but this probably does not matter since all roms are extracted from zip archives.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #11 on: 16 April 2023, 16:28 »

rezip...yikes...that of course unzips each file from each archive to temp and recompresses the files....why do you do that? If you want zip files in your destination, simply use "zip"....if something decompressed, something in a rar or 7z matches in the source it will be (unpacked) and repacked to zip....everything which is already zipped in the source can be copied without unpack/repacking....you're wasting time with rezip ;-)
« Last Edit: 16 April 2023, 16:30 by Roman »
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #12 on: 16 April 2023, 16:48 »

It's intentional - I'm not sure if source zip file is checked against any errors so I need everything rezipped. Anyway 46 % done and about 2 hours and 40 mins passed.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #13 on: 16 April 2023, 16:52 »

I pretty sure  a recursive commandline unzip -t on all archives afterwards would be faster ;-) but ok...
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #14 on: 16 April 2023, 17:12 »

ok...looks like other people also give some feedback:

a complete 253 (roms + chds) split sets to "mode: full" rebuild, "compress: zip", "sha: none", on one and the same HD without deletion of source files: 3441,69 seconds....which is 57 minutes...

...but "use links: hard" was used....so...a little bit of cheating :-) but still...remarkable
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #15 on: 16 April 2023, 18:18 »

Whole process started exactly 16:00:00. 7:58 took scan rebuild process was ended exactly at 20:16:10.

Now let's see how it will take with use of clrmame.

« Last Edit: 16 April 2023, 18:22 by haynor666 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #16 on: 16 April 2023, 18:59 »

Restart your machine first, otherwise you will cheat using the disk cache ;-)
...and of course you need to enable "recompress" in cmpro's rebuilder... (which corresponds to 'rezip')
Would be interesting to see....I mean with turning on "rezip" you of course disabled the major speed optimization....since it doesn't copy full archives in that case
« Last Edit: 16 April 2023, 19:12 by Roman »
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #17 on: 17 April 2023, 10:39 »

Rebuilded from the same source. Started at 7:15 ended about 12:35 so more than 5 hours was needed for entire process :)
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #18 on: 17 April 2023, 11:02 »

Nice to hear...that old cmpro is slower ;-) I was a bit sceptical since you turned off all of the major speed advantages....but sounds good to me....

Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Re: rebuilder 0.05 released
« Reply #19 on: 17 April 2023, 11:16 »

Worth to mention that some zip archives have different crcs (for example 1941.zip) but size and files inside are exactly the same. Most of zips are exactly the same. I assume that clrmamepro adds some files into zips with different order.
Logged
Pages: [1] 2   Go Up
 

Page created in 0.154 seconds with 20 queries.

anything