101
clrmame Discussion / Re: Rebuilder 0.06 released
« on: 07 May 2023, 18:27 »
- 23 April 2024, 08:31
- Welcome, Guest
News:
The new forum is online, hope you enjoy it!
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
102
clrmame Discussion / Re: Rebuilder 0.06 released
« on: 07 May 2023, 17:57 »
Not really a realistic scenario, since I don't know anyone who does a) a full rebuild and having b) compress: rezip and sha1:both enabled. Such options are timekillers (beside the amout of a rebuild).
As an example, if you don't use rezip, sha1 none and hardlinks, your full rebuild is most likely done within an hour (depending a bit from where to where you operate)
I know the argument "but I want to be 100% sure that the archives are fine"...but actually a post run commandline based archive test is faster. Same can apply for "but I want identically compressed archives".
The typical use case for the rebuilder is to update an existing collection with a bunch of sets.
But of course, if you like to spend 4.25 hours to always redo everything, ok...it's of course doable ;-)
As an example, if you don't use rezip, sha1 none and hardlinks, your full rebuild is most likely done within an hour (depending a bit from where to where you operate)
I know the argument "but I want to be 100% sure that the archives are fine"...but actually a post run commandline based archive test is faster. Same can apply for "but I want identically compressed archives".
The typical use case for the rebuilder is to update an existing collection with a bunch of sets.
But of course, if you like to spend 4.25 hours to always redo everything, ok...it's of course doable ;-)
103
clrmame Discussion / Re: Rebuilder 0.06 released
« on: 05 May 2023, 04:41 »
….and now compare it to old cmpro :-)
You have to be careful when comparing results...the diskcache can fool you immensly. So you need to compare either both with or without diskcache...
So the best ist (besides using the same source/destination) a reboot of the machine in between the tests and doing 1 scan.
Or you run multiple ones in a row for one version and take an average.
I'd expect something about 1 to 30% speed gain compared to 0.05...but yes, it also depends on your machine/number of cores
You have to be careful when comparing results...the diskcache can fool you immensly. So you need to compare either both with or without diskcache...
So the best ist (besides using the same source/destination) a reboot of the machine in between the tests and doing 1 scan.
Or you run multiple ones in a row for one version and take an average.
I'd expect something about 1 to 30% speed gain compared to 0.05...but yes, it also depends on your machine/number of cores
104
clrmame Discussion / Rebuilder 0.06 released
« on: 04 May 2023, 18:35 »
Rebuilder 0.06 released
https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v006.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html
Doing source and destination matches now in multiple threads...gives a nice speed boost.
https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v006.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html
Doing source and destination matches now in multiple threads...gives a nice speed boost.
105
clrmame Discussion / Re: How to rebuild romset without added clones?
« on: 24 April 2023, 05:59 »
The set merge modes (standalone, new rebuilder tool), full (merged), split (merged), un/not-merged don't play a role here. They only define how the sets are stored on your system. If you want to "cherry pick" several sets, you need to disable the ones you don't want. In old clrmamepro, you can do this in Scanner->Scan Results Tree window, bottom left 'set information' button. In Set Information you have various ways to enable/disable sets either by single-ticking, regular expressions, filter lists and so on. Only enabled ones will be rebuilt.
In the new rebuilder tool you can set a regular expression via the -f --filter parameter.
In the new rebuilder tool you can set a regular expression via the -f --filter parameter.
107
clrmame Discussion / Re: -listxml VS hash/*.xml
« on: 22 April 2023, 18:59 »
Especially the rompath setup is really really annoying which I have to admit....but as I said, this might change drastically.
Actually I don't know the reason why -listsoftware does not include all software lists of the hash folder. Well, at least in the past, several were not included in the extract. And since normally only a handful of lists change from MAME version to version, it makes more sense to have a profiler per software lists. Rompath generation and assignement can be here done via the batcher...
Actually I don't know the reason why -listsoftware does not include all software lists of the hash folder. Well, at least in the past, several were not included in the extract. And since normally only a handful of lists change from MAME version to version, it makes more sense to have a profiler per software lists. Rompath generation and assignement can be here done via the batcher...
108
clrmame Discussion / Re: rebuilder 0.05 released
« on: 22 April 2023, 18:41 »
There is a difference of parallel accessing the disk or doing parallel calculate threads….it needs a bit of tuning and checking to get a good result. If you force too many threads your cpu needs time for swapping etc… so as mentioned I will play around a bit. Dont expect too much….you also produce overhead with starting threads
109
clrmame Discussion / Re: -listxml VS hash/*.xml
« on: 22 April 2023, 05:40 »
When you import your data from a mame.exe you're asked after the import (unless the belonging option is unticked) if you want to import software lists. Then cmpro does load this -listsoftware data and you got all software lists in one profile. But....first of all, -listsoftware does not include all software lists from mame's hashfolder and secondly, it's a real pain to set this mode up correctly. You'd need unique distinct rompaths for each software list and scanning takes long, more memory usage etc..etc...This mode is not recommended.
Hopefully such problems will be history with a new scanner. You see that the new rebuilder already handles -listsoftware flawlessly and efficient.
Hopefully such problems will be history with a new scanner. You see that the new rebuilder already handles -listsoftware flawlessly and efficient.
110
clrmame Discussion / Re: rebuilder 0.05 released
« on: 21 April 2023, 06:32 »
As you (the audience) might have noticed, "fool-for-a-day, troll-for-a-life-time" Jessica Jones, rolled over the boards again and was removed by admins....nothing unusual...this happens once a year...
But the really funny part is:
The discussion gave me an idea where threading can give some extra speed up....It's not related to the troll comments at all (CUDA for SHA1 etc...)
But a first small test was already pretty positive which gave a extra ~25% speed boost when sha1 input scanning was enabled....
For non sha1 runs it's not that big since you simply have pure lookups without decompression/sha1 and the actual gain by multiple threads is eaten up by the overhead to create the threads...
I will check how this scales for larger data and for number of threads....
So..Thank you Jessica Jones...you made my day. See you next year.
But the really funny part is:
The discussion gave me an idea where threading can give some extra speed up....It's not related to the troll comments at all (CUDA for SHA1 etc...)
But a first small test was already pretty positive which gave a extra ~25% speed boost when sha1 input scanning was enabled....
For non sha1 runs it's not that big since you simply have pure lookups without decompression/sha1 and the actual gain by multiple threads is eaten up by the overhead to create the threads...
I will check how this scales for larger data and for number of threads....
So..Thank you Jessica Jones...you made my day. See you next year.
111
clrmame Discussion / Re: rebuilder 0.05 released
« on: 17 April 2023, 11:24 »
Hash values of an archive is of course obsolete.....order of added files and compression rate of course alters it....there is no guranteed add order, no sort, no nothing
112
clrmame Discussion / Re: rebuilder 0.05 released
« on: 17 April 2023, 11:02 »
Nice to hear...that old cmpro is slower ;-) I was a bit sceptical since you turned off all of the major speed advantages....but sounds good to me....
113
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 18:59 »
Restart your machine first, otherwise you will cheat using the disk cache ;-)
...and of course you need to enable "recompress" in cmpro's rebuilder... (which corresponds to 'rezip')
Would be interesting to see....I mean with turning on "rezip" you of course disabled the major speed optimization....since it doesn't copy full archives in that case
...and of course you need to enable "recompress" in cmpro's rebuilder... (which corresponds to 'rezip')
Would be interesting to see....I mean with turning on "rezip" you of course disabled the major speed optimization....since it doesn't copy full archives in that case
114
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 17:12 »
ok...looks like other people also give some feedback:
a complete 253 (roms + chds) split sets to "mode: full" rebuild, "compress: zip", "sha: none", on one and the same HD without deletion of source files: 3441,69 seconds....which is 57 minutes...
...but "use links: hard" was used....so...a little bit of cheating :-) but still...remarkable
a complete 253 (roms + chds) split sets to "mode: full" rebuild, "compress: zip", "sha: none", on one and the same HD without deletion of source files: 3441,69 seconds....which is 57 minutes...
...but "use links: hard" was used....so...a little bit of cheating :-) but still...remarkable
115
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 16:52 »
I pretty sure a recursive commandline unzip -t on all archives afterwards would be faster ;-) but ok...
116
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 16:28 »
rezip...yikes...that of course unzips each file from each archive to temp and recompresses the files....why do you do that? If you want zip files in your destination, simply use "zip"....if something decompressed, something in a rar or 7z matches in the source it will be (unpacked) and repacked to zip....everything which is already zipped in the source can be copied without unpack/repacking....you're wasting time with rezip ;-)
117
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 15:38 »
hmm...all mechanial ones (3 chds, ~15k sets), old cmpro (without chds): 210 seconds, new rebuilder (with chds): 109 seconds...
(split, no sha1, zip2zip, hd to (different) hd)
(split, no sha1, zip2zip, hd to (different) hd)
118
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 16 April 2023, 15:11 »
no problem...such things do happen..
119
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 14:40 »
Don't forget, the new rebuilder rebuilds CHDS, too.....and multi gb chds can take a while to copy :-) (ever tried it manually .... your system will be pretty busy)
I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace
I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace
120
clrmame Discussion / Re: mame merged sets not scanning correctly?
« on: 16 April 2023, 13:45 »Quote
As for CMP, (unless I'm doing something complete wrong here) it looks like even Rebuilder 0.05 can't produce a merged mame romset that can satisfy the scanner.. (no offence).Got several testers on board and noone reported a problem. So I guess you may have some setting wrongly set or you're using a wrong datasource (as reported before 239 vs 253)
Quote
how is it possible that alpinesa isn't listed as a clone to its parent alpines. in doing so, CMP wrongfully considers alpinesa as the parent and wants to
In general you can't fully merge alpinesa in alpines since they share an identically named rom with different hash (af2ver-a_ll.ic2). When cmpro imports data, it will prompt you about it (you most likely clicked on "ok to all" when the first of such events happend).
The hash collision detection will add a subfolder for it.... and it will show up as in the screenshot below.
Since you seem to have some weird effects which can not be repeated, it would be best if you send me your cmpro.ini, and the used .cmp file (cmpro's settings folder) and the used datfile (if any, if not, state which exe you're using).