EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - Roman

Pages: [1] 2 3 4 5 6 ... 12
1
clrmame Discussion / wip
« on: 05 February 2025, 11:57 »
Ok, before really starting on the profiler....I'm bringing up some user feedback here....

- when does "auto" thread selection becomes a problem? There seems to be cases where an auto selection picks the highest available thread number available on the system (which is intended on compressed files), however on a NAS this seems to cause seeking and transfering slowdowns....While on a local drive 28 threads seem to work fine with compressed sets I wonder if auto should also have some kind of max value...maybe 28...maybe less. Surely you can select a fixed value anyhow. Maybe users already played around a bit and have an idea...

- zstd, well, a topic which is not that new. Current situation: I'm using ZipArchive as 3rd party library for zip files. Why? Because it has a nice feature, the direct copy of compressed data without the need of recompressing it. This is a reason why rebuilding is fast when working with zipfiles. For 7z (reading/writing) and rar (reading) I use bit7z which is a lib on top of 7z.dll. It of course also supports zip. Even zip where files are compressed with zstd compression mode.
So to support zstd, we could wait till ZipArchive added it. I've asked the author over a year ago already.....but he seems busy with real life which is good. Now bit7z does support reading and writing of zstd. So why not switching to it? Well, because of the upper mentioned direct copy option.
So what is possible: I could add an option that you can switch completely to bit7z, supporting zstd. The direct copy option is then not possible and you most likely see a speed drop when it comes to adding files. The direct question then is: What happens when the rebuilder creates a new file? What will be the compression method then? Will it be zstd by default? And what if the rebuilder adds a file to an existing non zstd zip archive? Will you end up with an archive with mixed compression modes.
Another possibility would be that when reading a zip file I check if it uses at least one zstd file and switch internally to bit7z....so the change happens invisble to the user.....but still you get the same questions as above
I will think a bit about it.

Little update already: Currently I don't see an option to specify zstd for adding files in bit7z....asking the author now....


ah yeah...by the way...there were some changes lately so here's the latest wip:

UI:
- fixed: remove datasource XML file when hash folder changed (should only happen when XML file was internally created from an exe export)

Core:
- added: "available:" filter which limits the machine selection to files you have. You might notice a count difference in total vs filtered even when you got all files. This is based on the fact that a) empty sets are excluded and b) there are clones which   are totally included in their parent, so -in split mode- you don't have a standalone file/folder for it. So don't worry about the count.
- misc:  "file:" filter, in case you work with softwarelist collections, you'd need to prefix single entries with sl-name#pacman (e.g. a2600#pacman) to specify which set you're refering to. Not needed for single sl files or standard dats though.
- misc:  dupe output now shows all found paths belonging to a machine
- misc:  some earlier cancel returns in fix wrong named disk/sample/rom/machine
- fixed: dupes can be listed multiple times

3
clrmame Discussion / clrmame v0.2 released
« on: 20 January 2025, 19:59 »
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_v02.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:
- added: #thread selector
- added: scanner, Clipboard->Copy Machine Names option
- fixed: rebuilder log Copy To Clipboard misses linefeed

Core:
- added: you can define a thread pool size which is used for parallel scanning rebuilding/etc. Either you keep auto (default value) or pick the number of threads used for the pool yourself. "auto" picks a high value when you work with compressed files and a low value when using decompressed ones. A low value is better when you run into file seek overhead issues. If you're not happy with auto, you can try to tweak it yourself.
- fixed: scanner and rebuilder ignore output path, mergemode, pattern changes when xml data hasn't changed
- misc: more effective reimplementation of thread pools and thread queueing
- misc: prefetching sha1s on an archive if needed (speed increase, espically for archives with lots of files)
- misc: rebuilder: major update / reworked internal logic, less complex, way smaller memory footprint especially for dats out of hell (dats with thousands of dupe files), faster.
- misc: scanner, smaller memory footprint and speed update for "dats out of hell"
- fixed: scanner, accidently getting a crc32 on sub folders showing an error
- fixed: scanner, detection of unnneeded empty sub folders failed
- fixed: scanner, using old archive name when trying to remove some unneeded files causing can't remove files/can't access messages
- misc:  updated bit7z to 4.0.9
- misc:  updated pugixml to 1.15
- misc:  use "-" instead of "_" for replacing illegal file chars (cmpro align)
- misc:  monitoring hash folder for -listsoftware exports. In case of a change, cached data is ignored and a new export is generated. This is more for users which update their hash folders during the development phase of MAME on a regular basis and don't want to recompile the binary.
- misc: use one version number, fixed a tooltip typo ;-)

4
clrmame Discussion / Happy New Year...
« on: 12 January 2025, 14:52 »
Well well well....first of all...Happy New Year...2025.....a little bit late but anyhow, I'm alive and working...

A new version is somewhere around the corner.....I've spent the last days on doing some major rework on the rebuilder. Why? Well, it really performed lousy for "dats out of hell"....
For example, the progetto icons datfile which has 34000 icons where roughly 5000 are dupes....but running such a colleciton through the rebuilder means that it finds ~5000 matches for each of the ~5000 files....the rebuilder in clrmame wasn't well designed for that....and the memory footprint went up very high....and of course the rebuilding time.

The good thing is: The new implementation is way better. Plus it's also faster than the old one.
Just to give an impression: old cmpro rebuilder on the icons: 1m57s
clrmame rebuilder (while being fine on other stuff it totally failed here): not worth mentioning....I quit after an hour ;-)
reworked clrmame rebuilder for the next version: 1m42s

I have some minor stuff on my list before doing a release...but here's already a little sneak peak:



UI:
- added #thread selector

Core:
- misc: major rebuilder update / reworked internal logic, less complex, way  smaller memory footprint especially for dats out of hell (dats with thousands of dupes), faster.
 
- misc: reimplementation of thread pools
 
- added: you can define a thread pool size which is used for parallel scanning rebuilding/etc. Either you keep auto (default value) or pick the number of threads used for the pool yourself. "auto" picks a high value when you work with compressed files and a low value when using decompressed ones. A low value is better when you run into file seek overhead issues. If you're not happy with auto, you can try to tweak it yourself.

- fixed: scanner and rebuilder ignore output path, mergemode, pattern changes when xml data hasn't changed
- fixed: scanner, accidently getting a crc32 on sub folders showing an error
- fixed: scanner, detection of unnneeded empty sub folders failed
- fixed: scanner, using old archive name when trying to remove some unneeded files causing can't remove files/can't access messages
- misc:  updated bit7z to 4.0.9
- misc:  use "-" instead of "_" for replacing illegal file chars (cmpro align)
- misc:  monitoring hash folder for -listsoftware exports. In case of a change, cached data is ignored and a new export is generated. This is more for users which update their hash folders during the development phase of MAME on a regular basis and don't want to recompile the binary.
- misc: use one version number ;-)


5
Sorry, no profiler news at the moment.
Oddi's remark about slowlyness when scanning progetto snaps keeps me a bit busy these days. Not only that I already found some hash lookups which can be optimized a bit, I wonder how threading can influence the scanning speed.
Now John IV would say: I'm getting 1 second for a full scan, so threading rocks. yes, it does...when you're on a sdd...when you're on a hdd, too many threads may lead to too many seek operations which may slow the process down.
In case of an unpacked progetto collection (we talk about > 300000 files), each file's crc32 needs to be calculated....which is a lot seeking, reading and calculating (even if the actual calc is quickly done). Using too many threads may slow down the process....so currently I will do some benchmarking what the best value is.....I keep you updated with the output. Even if I see that too many threads slow it down, I somehow need a way to determine a good value (ok, I can show an input box where you simply define it as last possible solution ;-))

6
clrmame Discussion / a nightly cmpro...
« on: 07 December 2024, 20:06 »
if anyone still use the old tool....

https://mamedev.emulab.it/clrmamepro/binaries/cmpro64_20241207.zip
(64bit exe only)

it only fixes a problem with not detecting unneeded files in a sample parent set..... The bug was in there since decades I think ;)

Definetly time to use the new tools ;-)

7
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_008_015_1.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:
misc: restore default window positions when positions get out of range
misc: increase display time of tooltips
misc: slightly regrouped ui elements
added: scanner, context menu option to restore old scan results on startup and selection

Core:
misc: updated spdlog to 1.15.0
misc: updated to 7zip 24.09
misc: don't reload xml and build up structures when they are already in memory and need no refresh
misc: align and share rebuilder/scanner common routines
added: scanner, samples, supporting flac
fixed: scanner, typo in wine script
fixed: scanner, internal sha1 check for files with identical crc32 but different sha1 values isn't run
fixed: scanner, samples, wrong named files (case check only) aren't fixed
fixed: scanner, samples, wrong named files (case check only) also appear as unneeded
fixed: scanner, samples, decompressed files are marked as unneeded
fixed: scanner, samples, machines which reference themselves via sampleOf are marked as unneeded in full mode
fixed: scanner, confusing 'can't remove/backup' messages in case of circular renames between different sets
fixed: scanner, confusing 'can't remove/backup' messages in case a wrong file with a right name is replaced with a fill in

8
clrmame Discussion / Sneak....
« on: 25 November 2024, 16:45 »
Well, this week a new MAME will most likely be released....so I will wait for that one to see if something weird happens or not....This month I was pretty busy with real life but I found a little time to do some changes. Mainly I was working on some code moves here and there, simply because rebuilder and scanner have a lot in common...so it was clear to make such functions commonly used instead of having them doubled (which was still a bit of a relict when they weren't merged). I also had a look at sample scanning which had some issues

Core:
- misc:  updated spdlog to 1.15.0
- misc:  updated to 7zip 24.08
- misc:  don't reload xml and build up structures when they are already in memory and need no refresh
- misc:  align and share rebuilder/scanner common routines
- fixed: scanner, internal sha1 check for files with identical crc32 but different sha1 values isn't run
- fixed: scanner, samples, wrong named files (case check only) aren't fixed
- fixed: scanner, samples, wrong named files (case check only) also appear as unneeded
- fixed: scanner, samples, decompressed files are marked as unneeded
- fixed: scanner, samples, machines which reference themselves via sampleOf are marked as unneeded in full mode
- added: scanner, samples, supporting flac

UI:
- misc:  restore default window positions when positions get out of range
- misc:  increase display time of tooltips
- misc:  slightly regrouped ui elements
- fixed: scanner, typo in wine script

I also have some nice things on the request list...(...and surely there is the profiler topic....)...will see what I can do...
To name 3 of the request (but don't count on them soon)
- restore scan results on startup / change of xml
- only keep 1 file instead of fix dat and scan result since they nearly contain the same information
- maybe have some tree like output in the rebuilder to show which machines were rebuilt

9
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_007_014.zip

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:

misc: scanner, contextmenu option to sort by name instead of description
misc: Linux/Wine, users can get data from a (linux) mame binary. A wrapper script is included (requires a chmod +x ./wrapper.sh once though)
misc: Linux/Wine, minor changes regarding line breaks or layout

Core:

misc: in case of a -listsoftware data basis (either exe export or dat), additional sl hashes are added from either the used exe's hash folder (prio 1) or HashFolder specified in the settings xml file (prio 2).
fixed: some absolute paths aren't made 32k path length aware, this can lead to sideeffects, e.g. when doing backups (esp. in Linux/Wine)
fixed: scanner, removal of matched files from addPath/backup only works when at least one file was missing
fixed: scanner, removal of unneeded files stopped after first error during a delete process so that other files in the queue were skipped
fixed: scanner, fixing wrong named files/folders which only differ by a character case change fails
fixed: scanner, unneeded sample files are detected but not reported or fixed
fixed: scanner, (rare) "can't read" error message when accidently testing a folder for being a chd
fixed: scanner, (rare) fixing wrong named files/folders which only differ by a character case but map to multiple machines keeps only 1 instance (e.g. 3DO->3do & 3dobios)
fixed: scanner, circular renames inside a machine throw rename errors and only get resolved in a second run

10
clrmame Discussion / Nightly.....
« on: 27 October 2024, 18:32 »
While playing around with the Linux MAME export I totally forgot to say what else is new in the nightly build:

https://mamedev.emulab.it/clrmamepro/binaries/clrmame_006_013_20241027.zip

UI:
- misc: scanner, contextmenu option to sort by name instead of description
- misc: Linux/Wine, users can get data from a (linux) mame binary. For this, a wrapper script is included. You need to do a chmod +x ./wrapper.sh once though
- misc: Linux/Wine, minor changes regaring line breaks or layout

Core:
- fixed: scanner, unneeded sample files are detected but not reported or fixed
- fixed: scanner, rare "can't read" error message when accidently testing a folder for being a chd
- misc:  in case of a -listsoftware data basis (either exe export or dat), additional sl hashes are added from either the
used exe's hash folder (prio 1) or HashFolder specified in the settings xml file (prio 2).

A big "thank you" goes out to Shoegazer for the linux testing and finally pointing me in the right direction to fix the (hopefully) last outstanding problem.

So what's next?
Well...the upcoming week will most likely bring a new MAME version. As usual, I will check the scanner/rebuilder core with it to see if something weird pops up. So expect a new 'official' release after the MAME release.

After that I hope I finally find time to start with the profiler....that's the short view...on the long view I was also thinking about a batcher (currently I wonder if there is something which a profiler can already do, i.e. loading and scanning multiple dats, but I also have some ideas for a batcher which would act like a scheduler where you add your scan/rebuild jobs in a queue like thing....time will tell...profiler comes first).
And yes....I'm still sure to release the source :-)

11
UI:
- misc:  SHIFT clicking the clear buttons clear the full combo box
- misc:  when context menu show info->size is enabled, also show the sum on software list level
- misc:  remember last used tab on close/start
- fixed: restore window positions incl. vertical break isn't always working

Core:
- misc:  set current folder back to application folder after scan/rebuild to avoid folder locking
- fixed: scanner, already moved "missing but fixable" unpacked files removes the moved file afterwards when they were also marked as unneeded in their source position
- fixed: scanner, a failed file operation on "missing but fixable" removes the file when they were also marked as unneeded in their source position
- fixed: obsolete error message when using file filters
- fixed: avoid 'can't read' ios_base::failbit set errors on files where the file length is smaller than the minimal buffer for the operation (e.g. id or header read)
- fixed: scanner, obsolete "can't access" messages during fix unneeded due to not correctly filtering already removed archives


https://mamedev.emulab.it/clrmamepro/binaries/clrmame_006_013.zip

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

12
clrmame Discussion / MAME .270 ...and the new tools
« on: 25 September 2024, 18:14 »
ok...a quick look at the new MAME might give you a "can't access scherrym" error during fixing....but that's only temporary...and seems to get resolved automatically...however, I will have a look at the source of this message tomorrow..until now: nothing critical....

13
Combined rebuilder/scanner in one commandline and one UI version, let's name it clrmame for now

Core:

misc: scanner, remove matches from AddPath option now removes ALL matches, no matter if they were just real fill-ins for a missing file or already existing in a rompath
misc: hiding sample specific problems while loading datfile in rebuilder and in scanner when no sample paths are set
misc: scanner, skip not accessible unneeded files (loglevel trace can show information about them)
fixed: using relative paths fails after first scan due to wrong current folder
fixed: when an elapsed time takes less than 0 seconds, no value as elapsed time is shown ;-)
fixed: hash calculation of empty files
fixed: scanner, some unneeded files won't be removed but aren't reported either when fix is enabled

UI:

added: scanner, context menu option to show clone names (in full merge mode only) which have issues right beside the set name up to 5, if there are more, you can see all in tooltips
misc: merged rebuilder and scanner, switchable via tabs, can run in parallel (unless path sharing issues are detected)



https://mamedev.emulab.it/clrmamepro/binaries/clrmame_005_012.zip

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

14
clrmame Discussion / Sneak Peak
« on: 11 September 2024, 19:01 »
Well...pretty quiet here recently?

Well, that doesn't mean nothing happens:


UI:

- added: scanner, context menu option to show clone names (in full merge mode only) which have issues right beside the set name
         up to 5, if there are more, you can see all in tooltips

Core:


- fixed: using relative paths fails after first scan due to wrong current folder
- fixed: when an elapsed time takes less than 0 seconds, no value as elapsed time is shown ;-)
- fixed: hash calculation of empty files
- fixed: scanner, some unneeded files won't be removed but aren't reported either when fix is enabled
- misc:  scanner, remove matches from AddPath option now removes ALL matches, no matter if they were just real fill-ins for a missing file or already existing in a rompath
- misc:  hiding sample specific problems while loading datfile in rebuilder and in scanner when no sample paths are set



ah well..that's not all.....there is an additional picture attached....

15
clrmame Discussion / scanner 0.04, rebuilder 0.11
« on: 30 August 2024, 07:17 »
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

https://mamedev.emulab.it/clrmamepro/binaries/scanner004_rebuilder011.zip


Core:

- added: scanner, option to remove fill-in files from addpaths (this can include backup when 'act as' option is on). Only files which were actually needed in the rompaths are removed
- misc:  changed elapsed time format and show an overall duration
- fixed: handling of loadflag="continue" fails in software list based xmls
- fixed: scanner, fixing missing roms/disks can accidently move the rom/disk when it shouldn't
- fixed: scanner, parentfolders of fill-in files can be touched (timestamp update) when they shouldn't
- fixed: scanner, fix missing roms/disks can accidently look in a folder where the match is not present leading to can't access/can't backup messages

UI:

- added: checkbox for upper mentioned remove fill-in files option
- misc: updated some flyovers




By the way: If you have a 0 byte gekimaka.zip in your rompath, you need to remove it manually first. That could have been created with the .03 scanner.....

16
clrmame Discussion / new scanner and MAME .269
« on: 29 August 2024, 06:49 »
End of month, a new MAME is out, a fantastic opportunity to find issues in the new tools...

a quick look at it shows me that there can be some "can't access" and "can't backup" messages around wontame, dokodemo, gekimaka sets.
Such sets have roms and chds and the missing files in the MAME.269 cycle can be taken from other sets (i.e. pingu). Unfortunately the new scanner accidently also takes the chd folder of pingu for fixing the roms...where it doesn't find the roms and reports the issues.

Working on a fix (choosing the right source (dir or archive) in case of multiple sources) now and I guess a new version will be out soon.

17
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

https://mamedev.emulab.it/clrmamepro/binaries/scanner003_rebuilder010.zip

Core:

- added: long path/filename support (32k)

- misc:  update to bit7z 4.0.8

- misc:  scanner: propagate changed file timestamp up to rom/samplepath level
- misc:  scanner: improved preferedPackMethod a bit
- misc:  scanner: whenever possible, move (instead of copy/delete) missing but fixable chds or decompressed roms

- fixed: erroneously allow \ as an machine name character

- fixed: scanner, stopping the scan during fix operations doesn't have an effect
- fixed: scanner, freeze when fixing unneeded decompressed files (or chds) (0.02.1)
- fixed: scanner, wrong named files which are also fill-ins for missing files can trigger an obsolete file removal operation (sdiamond MAME.268)
- fixed: scanner, unneeded files/folders contain multiple identical entries (aa3020 MAME.268)
- fixed: scanner, fixing a wrong named folder failed when additional pattern folders were involved (copy error)
- fixed: scanner, fixing a wrong named folder failed when folder can have multiple new names (aa3020/a3010/aa5000 MAME.268)  (move error)
- fixed: scanner, prefered path lookup for software list collections never returned additional pattern folders (copy error)
- fixed: scanner, prefered path lookup for wrong named machines which only consist of chds failed (freeze during backup)
- fixed: scanner, removal of unneeded files can result in an empty folder which isn't removed

UI:

- added: scanner, always have software list collection sublevels (before a complete sl list only showed the complete top level)
- added: scanner, context menu option to auto expand scan results tree
- added: scanner, context menu list options are now also available on software list level

- fixed: active paths are not always remembered correctly when switching xmls or on initial load

- fixed: scanner, empty sofware lists show complete instead of empty icon
- fixed: scanner, scan results tree/log split isn't refreshed after minimizing/maximizing window




and in case you're interested in old cmpro:

4.049

fixed: falsely remove backup's _unknown folder in case of a failed backup which might remove already backuped and removed unneeded files. Only if _unknown folder isn't present before.
fixed: skip a non software list rompath if it is also assigned to a non active software list which would list all sets in that path as fully missing (combined mode only)
fixed: missing disk merging information within softwarelists, so that some identical chds stay in the clone set (combined mode only)
added: verify delta chds (via chdman)

18
clrmame Discussion / New scanner 0.02.1 out
« on: 15 July 2024, 19:43 »
UI:

- added: scanner log *and* tree can be X/Y resized now (mouse click+move between the 2 controls)
- added: clear button to clear all combo boxes for current 'profile' (rebuilder and scanner)
- added: clear button to remove not existing XML/EXE entries
- misc: increased the number of remembered xml entries to 10 (rebuilder and scanner)
- added: controls for unnneeded masks, include backup as addpath
- misc: improved storing of per-xml data
- fixed: filtermode setting isn't restored


Core:

- added: support more XML based datfiles
- added: unneeded masks (regular expression to filter out specifc unneeded files/folders) (e.g. ^.*\\(CHD|SL)$)
- added: handle backup as addpath option
- added: filter option supports prefix 'file:' to specify a file holding setnames used for filtering
- fixed: several obsolete "can't remove" errors based on not taking an in between rename into account
- fixed: machine/rom/disk/sample names may contain illegal path characters (incl. ending with .). Remove characters and warn about it
- fixed: prevent rom/disk with identical names but different hashes to get overwritten in decompressed standalone or full merged mode (e.g. sgi_mips chds). Filename gets prefixed with hash. MAME might fail loading it but at least the file is not lost
- fixed: fix-missing can skip one file from a batch and doesn't fix or report it
- fixed: application quits when directory scans cause an exception (e.g. access right denied)

ReadMe

Download

Feel free to support me


Update: 0.02.1 fixes a freeze when fixing unneeded decompressed files....

19
clrmame Discussion / Forum Spam Bot Protection
« on: 07 July 2024, 19:47 »
Looks like the forum reacts a bit weirdly and more strictly to posts today....I also had problems getting a reply through....used a different browser and it worked though.
Already informed the forum owner about it...maybe some software update causes this...


Sometimes logging out and in again helps, too.....

20
clrmame Discussion / MAME .267 and the new scanner
« on: 30 June 2024, 12:19 »
So new MAME is finally out....time to check the scanner on it... and yes, there are some things I need to look into.

Trivia (Questions Series 7, alt question ROM) [folder: gtsers7a - parent: gtsers1]
Here either cmpro or the new scanner complains about the name. The rom name ends with a "." which is not possible under Windows...cmpro removes it from the name, the new scanner seems to keep it, so you end in a rename loop if you scan with cmpro then with the new scanner and so on...I will fix that

cmpro also complains about some missing clone roms in cmv4zga, gtsers7a ,mspacmane , mspacmane2, pleiadsgmp and pacmanmr while the new scanner doesn't do that in its first run. So you may have the feeling everything is ok, while a 2nd scan reveals the "fixable but missing" ones. They also get fixed, but they should get fixed in the first run already OR it should be listed.

I will definetly check on those and what the reason for them is....so, if you find other things besides of these sets, let me know

Pages: [1] 2 3 4 5 6 ... 12

Page created in 0.075 seconds with 18 queries.