EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: Rebuilder 0.06 released  (Read 2724 times)

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 112.0.0.0 Chrome 112.0.0.0
    • View Profile
Rebuilder 0.06 released
« on: 04 May 2023, 18:35 »

Rebuilder 0.06 released

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v006.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

Doing source and destination matches now in multiple threads...gives a nice speed boost.
Logged


ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 113.0.0.0 Chrome 113.0.0.0
    • View Profile
Re: Rebuilder 0.06 released
« Reply #1 on: 04 May 2023, 19:55 »

Thank you Roman, will test its performance ASAP

Unrelated: should the rebuilder deserve its own thread now? I do understand it's the "next cmp" so to speak, but still... :D
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 113.0.0.0 Chrome 113.0.0.0
    • View Profile
Re: Rebuilder 0.06 released
« Reply #2 on: 04 May 2023, 21:22 »

Hmmmmm "a nice speed boost" you say? Man I've just got a 234% speed increase!!!
I've reprocessed my latest full update job, and it took far less than half the time it took with rebuilder v0.05 (510s vs. 1194s. 42% of the time)
Whatever magic you've applied to Rebuilder v0.06, that's the way to go!!!  8)
« Last Edit: 04 May 2023, 21:23 by ntt3 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Mac OS X Mac OS X
  • Browser:
  • Safari 16.4 Safari 16.4
    • View Profile
Re: Rebuilder 0.06 released
« Reply #3 on: 05 May 2023, 04:41 »

….and now compare it to old cmpro :-)

You have to be careful when comparing results...the diskcache can fool you immensly. So you need to compare either both with or without diskcache...
So the best ist (besides using the same source/destination) a reboot of the machine in between the tests and doing 1 scan.
Or you run multiple ones in a row for one version and take an average.

I'd expect something about 1 to 30% speed gain compared to 0.05...but yes, it also depends on your machine/number of cores
« Last Edit: 05 May 2023, 05:44 by Roman »
Logged

ntt3

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 26
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 113.0.0.0 Chrome 113.0.0.0
    • View Profile
Re: Rebuilder 0.06 released
« Reply #4 on: 05 May 2023, 08:57 »

No caching taking place, rest assured, multiple runs take consistently around 500-510s with reboots in between
It should be noted that it's a powerful machine (12 core/24 thread, 64GB RAM, different SSD's for rebuilder source and target), but still the same I was using with rebuilder v0.05
The easy consideration would be that the new rebuilder scales linearly to the number of threads, but I suspect the relative performances of the source and target SSDs may play a role as well
Regardless, it's a huge improvement - at least for me :D
« Last Edit: 05 May 2023, 10:34 by ntt3 »
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 113.0.0.0 Chrome 113.0.0.0
    • View Profile
Re: Rebuilder 0.06 released
« Reply #5 on: 07 May 2023, 17:32 »

This time for version 254 and archives in zips so test differs from previous but I think most people will rebuild from zip files so it's more realistic scenario.

« Last Edit: 07 May 2023, 17:33 by haynor666 »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 113.0.0.0 Chrome 113.0.0.0
    • View Profile
Re: Rebuilder 0.06 released
« Reply #6 on: 07 May 2023, 17:57 »

Not really a realistic scenario, since I don't know anyone who does a) a full rebuild and having b) compress: rezip and sha1:both enabled. Such options are timekillers (beside the amout of a rebuild).
As an example, if you don't use rezip, sha1 none and hardlinks, your full rebuild is most likely done within an hour (depending a bit from where to where you operate)
I know the argument "but I want to be 100% sure that the archives are fine"...but actually a post run commandline based archive test is faster. Same can apply for "but I want identically compressed archives".
The typical use case for the rebuilder is to update an existing collection with a bunch of sets.

But of course, if you like to spend 4.25 hours to always redo everything, ok...it's of course doable ;-)
Logged

haynor666

  • retro maniac
  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 64
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 113.0.0.0 Chrome 113.0.0.0
    • View Profile
Re: Rebuilder 0.06 released
« Reply #7 on: 07 May 2023, 18:26 »

More realistic versus my previous tests ;)

Anyway it's improvement comparing to version 005.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 113.0.0.0 Chrome 113.0.0.0
    • View Profile
Re: Rebuilder 0.06 released
« Reply #8 on: 07 May 2023, 18:27 »

 :)
Logged
Pages: [1]   Go Up
 

Page created in 0.222 seconds with 19 queries.

anything
anything