EMULAB Forum
clrmamepro [English] => clrmame Discussion => Topic started by: belegdol on 28 July 2023, 16:57
-
Hello,
handling progetto-SNAPs DAT files has become increasingly painful over time. I have the zipped files, snap.zip and titles.zip stored in a SMB mount, each in its own rompath. Until now it was slow-ish but manageable, however attempting to scan 0.254 zip with 0.256 DAT file (pS_Snap_NS_20230728_(cm).zip) in preparation for a rebuild has not completed for over an hour. I already have missing check disabled, same for the deeper check. I have now moved the file to a local SSD just to see if the network is the culprit but the scan has been going on for several minutes now. The last sign of life is the query whether I want to delete the incorrectly sized files.
Should the files be stored unzipped or zipped? [1] says that speed is gained by not compressing but [2] says that unpacked sets are slow as the hashes need to be recalculated instead of being picked up from the zipfile itself. Thanks for the input.
[1] https://www.emulab.it/forum/index.php?topic=4112.0
[2] https://www.emulab.it/forum/index.php?topic=4055.0
-
I‘d go with decompressed sets and anything fixmissing related turned off. Use the rebuilder to add the update sets to the existing collection.
Future might bring something faster…..cough
-
I tried with unzipped sets but generating hashes from a NAS SMB share was very slow. I then tried with zipped again. Titles set got processed within minutes but the ingame one took a few hours. Ultimately I believe this was due to the "no snapshot available" PNG getting updated and ~20k out of ~45k PNGs being missing.
-
Yep placeholder files which are shared thousands of times are a killer. They are one of the reasons why fix missing options (and deeper look for fix missing etc) should be turned off. If you then also use solid 7z archives you can go on vacation while fixing 😁
Maybe a full rebuild is faster in such cases or a manual zip after working with decompressed sets….
As mentioned, maybe the not too far future brings you something faster 😁 however progetto snaps are not in the focus yet at all but I guess I will have a look at it too
-
I already had fix missing and deeper check turned off. My first attempts were even with check missing turned off, but it did not make much of a difference. It appears that there is another option being hit by the placeholder files disproportionately.
I think next time I will indeed rebuild the archive manually. Or at least read the snaps changelog to be aware what is coming :D