1
clrmame Discussion / wip
« on: 05 February 2025, 11:57 »
Ok, before really starting on the profiler....I'm bringing up some user feedback here....
- when does "auto" thread selection becomes a problem? There seems to be cases where an auto selection picks the highest available thread number available on the system (which is intended on compressed files), however on a NAS this seems to cause seeking and transfering slowdowns....While on a local drive 28 threads seem to work fine with compressed sets I wonder if auto should also have some kind of max value...maybe 28...maybe less. Surely you can select a fixed value anyhow. Maybe users already played around a bit and have an idea...
- zstd, well, a topic which is not that new. Current situation: I'm using ZipArchive as 3rd party library for zip files. Why? Because it has a nice feature, the direct copy of compressed data without the need of recompressing it. This is a reason why rebuilding is fast when working with zipfiles. For 7z (reading/writing) and rar (reading) I use bit7z which is a lib on top of 7z.dll. It of course also supports zip. Even zip where files are compressed with zstd compression mode.
So to support zstd, we could wait till ZipArchive added it. I've asked the author over a year ago already.....but he seems busy with real life which is good. Now bit7z does support reading and writing of zstd. So why not switching to it? Well, because of the upper mentioned direct copy option.
So what is possible: I could add an option that you can switch completely to bit7z, supporting zstd. The direct copy option is then not possible and you most likely see a speed drop when it comes to adding files. The direct question then is: What happens when the rebuilder creates a new file? What will be the compression method then? Will it be zstd by default? And what if the rebuilder adds a file to an existing non zstd zip archive? Will you end up with an archive with mixed compression modes.
Another possibility would be that when reading a zip file I check if it uses at least one zstd file and switch internally to bit7z....so the change happens invisble to the user.....but still you get the same questions as above
I will think a bit about it.
Little update already: Currently I don't see an option to specify zstd for adding files in bit7z....asking the author now....
ah yeah...by the way...there were some changes lately so here's the latest wip:
UI:
- fixed: remove datasource XML file when hash folder changed (should only happen when XML file was internally created from an exe export)
Core:
- added: "available:" filter which limits the machine selection to files you have. You might notice a count difference in total vs filtered even when you got all files. This is based on the fact that a) empty sets are excluded and b) there are clones which are totally included in their parent, so -in split mode- you don't have a standalone file/folder for it. So don't worry about the count.
- misc: "file:" filter, in case you work with softwarelist collections, you'd need to prefix single entries with sl-name#pacman (e.g. a2600#pacman) to specify which set you're refering to. Not needed for single sl files or standard dats though.
- misc: dupe output now shows all found paths belonging to a machine
- misc: some earlier cancel returns in fix wrong named disk/sample/rom/machine
- fixed: dupes can be listed multiple times
- when does "auto" thread selection becomes a problem? There seems to be cases where an auto selection picks the highest available thread number available on the system (which is intended on compressed files), however on a NAS this seems to cause seeking and transfering slowdowns....While on a local drive 28 threads seem to work fine with compressed sets I wonder if auto should also have some kind of max value...maybe 28...maybe less. Surely you can select a fixed value anyhow. Maybe users already played around a bit and have an idea...
- zstd, well, a topic which is not that new. Current situation: I'm using ZipArchive as 3rd party library for zip files. Why? Because it has a nice feature, the direct copy of compressed data without the need of recompressing it. This is a reason why rebuilding is fast when working with zipfiles. For 7z (reading/writing) and rar (reading) I use bit7z which is a lib on top of 7z.dll. It of course also supports zip. Even zip where files are compressed with zstd compression mode.
So to support zstd, we could wait till ZipArchive added it. I've asked the author over a year ago already.....but he seems busy with real life which is good. Now bit7z does support reading and writing of zstd. So why not switching to it? Well, because of the upper mentioned direct copy option.
So what is possible: I could add an option that you can switch completely to bit7z, supporting zstd. The direct copy option is then not possible and you most likely see a speed drop when it comes to adding files. The direct question then is: What happens when the rebuilder creates a new file? What will be the compression method then? Will it be zstd by default? And what if the rebuilder adds a file to an existing non zstd zip archive? Will you end up with an archive with mixed compression modes.
Another possibility would be that when reading a zip file I check if it uses at least one zstd file and switch internally to bit7z....so the change happens invisble to the user.....but still you get the same questions as above
I will think a bit about it.
Little update already: Currently I don't see an option to specify zstd for adding files in bit7z....asking the author now....
ah yeah...by the way...there were some changes lately so here's the latest wip:
UI:
- fixed: remove datasource XML file when hash folder changed (should only happen when XML file was internally created from an exe export)
Core:
- added: "available:" filter which limits the machine selection to files you have. You might notice a count difference in total vs filtered even when you got all files. This is based on the fact that a) empty sets are excluded and b) there are clones which are totally included in their parent, so -in split mode- you don't have a standalone file/folder for it. So don't worry about the count.
- misc: "file:" filter, in case you work with softwarelist collections, you'd need to prefix single entries with sl-name#pacman (e.g. a2600#pacman) to specify which set you're refering to. Not needed for single sl files or standard dats though.
- misc: dupe output now shows all found paths belonging to a machine
- misc: some earlier cancel returns in fix wrong named disk/sample/rom/machine
- fixed: dupes can be listed multiple times