EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - Roman

Pages: 1 ... 3 4 5 6 7 [8] 9 10 11 12 13 ... 167
141
clrmame Discussion / Re: rebuilder 0.05 released
« on: 22 April 2023, 18:41 »
There is a difference of parallel accessing the disk or doing parallel calculate threads….it needs a bit of tuning and checking to get a good result. If you force too many threads your cpu needs time for swapping etc… so as mentioned I will play around a bit. Dont expect too much….you also produce overhead with starting threads

142
clrmame Discussion / Re: -listxml VS hash/*.xml
« on: 22 April 2023, 05:40 »
When you import your data from a mame.exe you're asked after the import (unless the belonging option is unticked) if you want to import software lists. Then cmpro does load this -listsoftware data and you got all software lists in one profile. But....first of all, -listsoftware does not include all software lists from mame's hashfolder and secondly, it's a real pain to set this mode up correctly. You'd need unique distinct rompaths for each software list and scanning takes long, more memory usage etc..etc...This mode is not recommended.
Hopefully such problems will be history with a new scanner. You see that the new rebuilder already handles -listsoftware flawlessly and efficient.

143
clrmame Discussion / Re: rebuilder 0.05 released
« on: 21 April 2023, 06:32 »
As you (the audience) might have noticed, "fool-for-a-day, troll-for-a-life-time" Jessica Jones, rolled over the boards again and was removed by admins....nothing unusual...this happens once a year...

But the really funny part is:

The discussion gave me an idea where threading can give some extra speed up....It's not related to the troll comments at all (CUDA for SHA1 etc...)

But a first small test was already pretty positive which gave a extra ~25% speed boost when sha1 input scanning was enabled....
For non sha1 runs it's not that big since you simply have pure lookups without decompression/sha1 and the actual gain by multiple threads is eaten up by the overhead to create the threads...

I will check how this scales for larger data and for number of threads....

So..Thank you Jessica Jones...you made my day. See you next year.

144
clrmame Discussion / Re: rebuilder 0.05 released
« on: 17 April 2023, 11:24 »
Hash values of an archive is of course obsolete.....order of added files and compression rate of course alters it....there is no guranteed add order, no sort, no nothing

145
clrmame Discussion / Re: rebuilder 0.05 released
« on: 17 April 2023, 11:02 »
Nice to hear...that old cmpro is slower ;-) I was a bit sceptical since you turned off all of the major speed advantages....but sounds good to me....


146
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 18:59 »
Restart your machine first, otherwise you will cheat using the disk cache ;-)
...and of course you need to enable "recompress" in cmpro's rebuilder... (which corresponds to 'rezip')
Would be interesting to see....I mean with turning on "rezip" you of course disabled the major speed optimization....since it doesn't copy full archives in that case

147
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 17:12 »
ok...looks like other people also give some feedback:

a complete 253 (roms + chds) split sets to "mode: full" rebuild, "compress: zip", "sha: none", on one and the same HD without deletion of source files: 3441,69 seconds....which is 57 minutes...

...but "use links: hard" was used....so...a little bit of cheating :-) but still...remarkable

148
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 16:52 »
I pretty sure  a recursive commandline unzip -t on all archives afterwards would be faster ;-) but ok...

149
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 16:28 »
rezip...yikes...that of course unzips each file from each archive to temp and recompresses the files....why do you do that? If you want zip files in your destination, simply use "zip"....if something decompressed, something in a rar or 7z matches in the source it will be (unpacked) and repacked to zip....everything which is already zipped in the source can be copied without unpack/repacking....you're wasting time with rezip ;-)

150
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 15:38 »
hmm...all mechanial ones (3 chds, ~15k sets), old cmpro (without chds): 210 seconds, new rebuilder (with chds): 109 seconds...
(split, no sha1, zip2zip, hd to (different) hd)

151
no problem...such things do happen..

152
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 14:40 »
Don't forget, the new rebuilder rebuilds CHDS, too.....and multi gb chds can take a while to copy :-)  (ever tried it manually .... your system will be pretty busy)

I mean...rebuilding a complete collection is not the usual thing to do...but either way: It will be faster than before (surely you have to count in the chds copies)...not to mention the way way way modern c++ core etc..etc... with hardlinks you'd even be able to keep full and split sets without taking twice the amount of diskspace

153
Quote
As for CMP, (unless I'm doing something complete wrong here) it looks like even Rebuilder 0.05 can't produce a merged mame romset that can satisfy the scanner.. (no offence).
Got several testers on board and noone reported a problem. So I guess you may have some setting wrongly set or you're using a wrong datasource (as reported before 239 vs 253)

Quote
how is it possible that alpinesa isn't listed as a clone to its parent alpines. in doing so, CMP wrongfully considers alpinesa as the parent and wants to

In general you can't fully merge alpinesa in alpines since they share an identically named rom with different hash (af2ver-a_ll.ic2). When cmpro imports data, it will prompt you about it (you most likely clicked on "ok to all" when the first of such events happend).
The hash collision detection will add a subfolder for it.... and it will show up as in the screenshot below.

Since you seem to have some weird effects which can not be repeated, it would be best if you send me your cmpro.ini, and the used .cmp file (cmpro's settings folder) and the used datfile (if any, if not, state which exe you're using).


154
Quote
I understand that CMP, unlike RV might want to duplicate roms shared between clones if there is no merge TAG present. but not honoring the filesize and CRC as described in the XML seems weird.. Where is it getting this from?

Easy answer: You're tring to scan a 253 collection with a MAME binary based on 239 (the window title of your screenshot shows it). Back then the hashes were different.

155
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 13:32 »
Thanks for new version. I'm in process of rebuilding whole MAME 253 set without CHDs from fully extracted archives into separated directories.

Extracted files are placed on M.2 disk, destination will be Seagate SV35 3TB disk (not too fast but at the moment I don'r have extra SSD for destination). Let see how long it will take on Core i5 10600KF and 16 GB RAM versus standard clrmamepro :)


Generally the speed should be way faster than cmpro since the rebuilder is capable of copying sets directly (if they are ready for it). It may look slower on the first glance since scanning the source takes a bit while old cmpro took one file from the source and then rebuilt it when possible and so on....
Slowing down factors are: sha1 checks (without it, it flies, however you should know what you rebuild....crc32 collisions are not that uncommon), not setting a good temporary folders (in case unpack operations are necessary) and finally....solid 7z files sha1 enabled....this is a real crawler

But for example a sha1-less split zip-to-zip rebuild of let's say all NeoGeo sets is 18 seconds....fast enough I think......


156
clrmame Discussion / Re: rebuilder 0.05 released
« on: 16 April 2023, 13:29 »
tried it, but how do I verify the output?
https://www.emulab.it/forum/index.php?topic=8905.msg25366#msg25366


The readme.md tells you:



Since there is no scanner yet, can I scan my rebuilt sets with clrmamepro?
Yes, but be aware of the following:

  • clrmamepro does not have the standalone merge mode (it only has not-merged + disable separate bios sets, but either way it won't match)
  • clrmamepro's profiler options to parse disk/rom merge tags and don't create dummy clones have to be enabled (that's the default by the way) since the new rebuilder only cares about merge attributes for checking relationships.
  • for full merged sets, clrmamepro's settings option "Hash Collision Name" mode has to be enabled (this is not enabled by default) and profiler option "naming pattern" should be "%f%1" (default). The new rebuilder always uses subpaths for clones/dependencies and that matches this behaviour. Keep in mind, this is only valid for compressed sets. Decompressed ones don't have subfolders, then don't enable the hash collision name mode.


157
clrmame Discussion / rebuilder 0.05 released
« on: 14 April 2023, 20:06 »
Rebuilder 0.05 released

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.

As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).

RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.

Hope you enjoy it...


158
then you should switch to new rebuilder's standalone mode...which really produces "unmerged" sets ;-)

159
4TB HDs are < 100 EUR ......I come from a time where I paid > 800 (rough estimation) for an Amiga HD......for 32 MB.....

160
"As you say, I think it will deem it as unneeded, and move it to rev3 where it is 'needed', if not present, and discard it from rev4. But I have not tested this."

Why should it be needed in rev3 and not vice versa? just because some string says revision 3 versus 4 and an xml ordering of 3 before 4....Sorry...but I don't follow this logic.... :-)

You have parent/clone relationships of totally different kinds in MAME...where not even the setnames are in common...or the manufacturer etc....and yes, there you might also find identical files spread in the clones only...how do you decide there to which clone they really belong....

Again...if you think your/RomVault's full merge is more right, feel free to use it....there is no standard, there is no right way, it's just interpretation.

Pages: 1 ... 3 4 5 6 7 [8] 9 10 11 12 13 ... 167

Page created in 0.142 seconds with 20 queries.