EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: How to best deal with progetto-SNAPs?  (Read 2744 times)

belegdol

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 19
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 115.0 Firefox 115.0
    • View Profile
How to best deal with progetto-SNAPs?
« on: 28 July 2023, 16:57 »

Hello,

handling progetto-SNAPs DAT files has become increasingly painful over time. I have the zipped files, snap.zip and titles.zip stored in a SMB mount, each in its own rompath. Until now it was slow-ish but manageable, however attempting to scan 0.254 zip with 0.256 DAT file (pS_Snap_NS_20230728_(cm).zip) in preparation for a rebuild has not completed for over an hour. I already have missing check disabled, same for the deeper check. I have now moved the file to a local SSD just to see if the network is the culprit but the scan has been going on for several minutes now. The last sign of life is the query whether I want to delete the incorrectly sized files.
Should the files be stored unzipped or zipped? [1] says that speed is gained by not compressing but [2] says that unpacked sets are slow as the hashes need to be recalculated instead of being picked up from the zipfile itself. Thanks for the input.

[1] https://www.emulab.it/forum/index.php?topic=4112.0
[2] https://www.emulab.it/forum/index.php?topic=4055.0
« Last Edit: 28 July 2023, 16:58 by belegdol »
Logged


Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Offline Offline
  • Posts: 3324
  • Operating System:
  • Mac OS X Mac OS X
  • Browser:
  • Safari 16.5.2 Safari 16.5.2
    • View Profile
Re: How to best deal with progetto-SNAPs?
« Reply #1 on: 28 July 2023, 17:52 »

I‘d go with decompressed sets and anything fixmissing related turned off. Use the rebuilder to add the update sets to the existing collection.

Future might bring something faster…..cough
« Last Edit: 28 July 2023, 17:58 by Roman »
Logged

belegdol

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 19
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 115.0 Firefox 115.0
    • View Profile
Re: How to best deal with progetto-SNAPs?
« Reply #2 on: 29 July 2023, 08:29 »

I tried with unzipped sets but generating hashes from a NAS SMB share was very slow. I then tried with zipped again. Titles set got processed within minutes but the ingame one took a few hours. Ultimately I believe this was due to the "no snapshot available" PNG getting updated and ~20k out of ~45k PNGs being missing.
« Last Edit: 29 July 2023, 08:31 by belegdol »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 118
  • Offline Offline
  • Posts: 3324
  • Operating System:
  • Mac OS X Mac OS X
  • Browser:
  • Safari 16.5.2 Safari 16.5.2
    • View Profile
Re: How to best deal with progetto-SNAPs?
« Reply #3 on: 29 July 2023, 09:18 »

Yep placeholder files which are shared thousands of times are a killer. They are one of the reasons why fix missing options (and deeper look for fix missing etc) should be turned off. If you then also use solid 7z archives you can go on vacation while fixing 😁

Maybe a full rebuild is faster in such cases or a manual zip after working with decompressed sets….

As mentioned, maybe the not too far future brings you something faster 😁 however progetto snaps are not in the focus yet at all but I guess I will have a look at it too
Logged

belegdol

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 19
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 115.0 Firefox 115.0
    • View Profile
Re: How to best deal with progetto-SNAPs?
« Reply #4 on: 29 July 2023, 10:35 »

I already had fix missing and deeper check turned off. My first attempts were even with check missing turned off, but it did not make much of a difference. It appears that there is another option being hit by the placeholder files disproportionately.
I think next time I will indeed rebuild the archive manually. Or at least read the snaps changelog to be aware what is coming :D
Logged
Pages: [1]   Go Up
 

Page created in 0.14 seconds with 23 queries.