EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - Roman

Pages: [1] 2 3 4 5 6 ... 12
1
clrmame Discussion / clrmame 0.4 released
« on: 20 April 2025, 17:01 »
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_v04.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:

- scanner, fixed showing of empty and complete machines
- scanner, fixed showing of non rom/sample/disk related machine/file/folder issues

Core:

- added: read/write support for zstandard compressed zip files
- rebuilder, two zstd compression methods added, newly created/added files   will use it (existing destination files which don't get replaced won't be   updated). Source files which don't use the set compression method won't be taken into account when checking for a direct archive copy
- scanner, automatically detects if you prefer zstd or deflate compressed zips and uses the preferred setting the case that new files are added.
- with both, rebuilder/scanner you can theoretically end up with zip files which have files in it with different compression methods. Rebuilder: if you
  rebuild to existing files, scanner, if you already have a mixture of archives using different compression methods.

- added: scanner, reporting 'wrong' chd version. Current expected version is 5, warning is not shown for baddumps. Version and baddump warning can be altered in settings.xml

- misc:  changed the behaviour of devices which have romOf dependencies. When fully merging such devices they are now handled like parent/clone machines. On the one hand it makes sense since there are merge attributes indicating that such sets belong together but there are also cases where they are distinct. This is now aligned with clrmamepro, but still something which might need further discussion

- misc:  updated cli11 to 2.5.0
- misc:  updated spdlog to 1.15.2
- misc:  updated bit7z to 4.0.10

- misc:  scanner, using absolute pathnames for error reporting in path scan
- misc:  scanner, only trace-log machines with issues in fixing phase
- fixed: scanner, typo "uneeded"
- fixed: scanner: detection of empty but unneeded folders in archives
- fixed: scanner, backup of chds in software list collections can fail
- fixed: scanner, archives which match a valid rompath subfolder name (e.g. softwarelist name via pattern or automatically) are falsely iterated like a folder instead of listing them as unneeded (wrong placed)
- fixed: crash on rom definitions without a crc
- fixed: filter enrichment might miss sampleof dependencies
- fixed: doing archive backups from a folder and backup archive already exists creates a new archive instead of merging files in

2
clrmame Discussion / Easter egg
« on: 11 April 2025, 18:51 »
93 (0x5D)


;-)

….coming to clrmameUI…so not an active easter egg yet

3
clrmame Discussion / Monthly sneak
« on: 07 April 2025, 20:08 »
Well...it's been a while so maybe you're interested what's happening behind the scenes...

well, first of all, I started with the profiler. Nothing to show/work at the moment since a) real life keeps me extemely busy and b) there are some other core tasks to do.

So here what's already in the next version:
UI:
- scanner, fixed showing of empty and complete machines
- scanner, fixed showing of non rom/sample/disk related machine issues (e.g. wrong named machine)

Core:
- added: scanner reports 'wrong' chd version. Current expected version is 5, warning is not shown for baddumps. Version and baddump warning can be altered in settings.xml
- misc:  updated cli11 to 2.5.0
- misc:  updated spdlog to 1.15.2
- misc:  using absolute pathnames for error reporting during path scan
- misc:  only list machines with issues as trace output in fixing phase
- fixed: typo "uneeded"
- fixed: filter enrichment might miss sampleof dependencies
- fixed: doing archive backups from a folder and backup archive already exists creates a new archive instead of merging files in
- fixed: archives which match a valid rompath subfolder name (e.g. softwarelist name via pattern or automatically) are falsely iterated like a folder instead of listing them as unneeded (wrong placed)


Still in the queue:

- trying to repeat the crash which 2 users reported  when fixing the Amiga SL 1000/3000 change (MAME 274-275). Unfortunately I can't repeat it at all at the moment...so if anyone is able to repeat it, please contact me (see https://www.emulab.it/forum/index.php?topic=9882.msg27364#msg27364)
- trying to find out how chds were placed in a samples folder (see https://www.emulab.it/forum/index.php?topic=9893.0)
- checking some WINE SHA1 checking warnings
- trying to align clrmamepro and clrmame/UI for devices which have a romof dependency and use or don't use merge attributes on files. Currently I think that clrmameUI does it right...but I need to compare and check...


4
clrmame Discussion / clrmame 0.3 released
« on: 26 February 2025, 19:26 »
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_v031.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:

added: optionally hide/show roms/disks/samples output via context menu
fixed: remove datasource XML file when hash folder changed (should only happen when XML file was internally created from an exe export)

Core:

added: "available:" filter which limits the machine selection to files you have. You might notice a count difference in total vs filtered even when you got all files. This is based on the fact that a) empty sets are excluded and b) there are clones which are totally included in their parent, so -in split mode- you don't have a standalone file/folder for it. So don't worry about the count.
misc: "file:" filter, in case you work with softwarelist collections, you'd need to prefix single entries with sl-name#pacman (e.g. a2600#pacman) to specify which set you're refering to. Not needed for single sl files or standard dats though.
misc: some earlier cancel returns in fix wrong named disk/sample/rom/machine
misc: dupe output now shows all found paths belonging to a machine
misc: updated spdlog to 1.15.1
misc: limited AUTO thread switch to 25 threads as max (or less depending on your hardware). You can still manually select more or less if you like
fixed: dupes can be listed multiple times
fixed: backups for unneeded folders can create wrong and very long folders due to wrong encapsuling
fixed: backup can miss empty unneeded folders at first scan
fixed: fill-in file from addpath/backup path was removed (when requested) even when fill-in copy failed and you might had a 0 byte new archive
fixed: log pattern in commandline mode

5
clrmame Discussion / forum posting problems
« on: 14 February 2025, 11:36 »
This forum uses some spam protection for some time which can cause problems when you try to post.
Since I'm not the operator of the forum, I can't do anything about it, but here are some hints which might help you:

When you see a "session timed out while posting" message, you can simply hit the post button again which usually refreshes the session and pushes your message.

When seeing a "CleanTalk: *** Forbidden. Please enable JavaScript. Browser seems to be spambot. Anti-Spam by CleanTalk. ***" message, you can either try to change your text or copy your written text, logout, clean cookies and relogin and add your text again. Usually your post is accepted then.

6
clrmame Discussion / wip
« on: 05 February 2025, 11:57 »
Ok, before really starting on the profiler....I'm bringing up some user feedback here....

- when does "auto" thread selection becomes a problem? There seems to be cases where an auto selection picks the highest available thread number available on the system (which is intended on compressed files), however on a NAS this seems to cause seeking and transfering slowdowns....While on a local drive 28 threads seem to work fine with compressed sets I wonder if auto should also have some kind of max value...maybe 28...maybe less. Surely you can select a fixed value anyhow. Maybe users already played around a bit and have an idea...

- zstd, well, a topic which is not that new. Current situation: I'm using ZipArchive as 3rd party library for zip files. Why? Because it has a nice feature, the direct copy of compressed data without the need of recompressing it. This is a reason why rebuilding is fast when working with zipfiles. For 7z (reading/writing) and rar (reading) I use bit7z which is a lib on top of 7z.dll. It of course also supports zip. Even zip where files are compressed with zstd compression mode.
So to support zstd, we could wait till ZipArchive added it. I've asked the author over a year ago already.....but he seems busy with real life which is good. Now bit7z does support reading and writing of zstd. So why not switching to it? Well, because of the upper mentioned direct copy option.
So what is possible: I could add an option that you can switch completely to bit7z, supporting zstd. The direct copy option is then not possible and you most likely see a speed drop when it comes to adding files. The direct question then is: What happens when the rebuilder creates a new file? What will be the compression method then? Will it be zstd by default? And what if the rebuilder adds a file to an existing non zstd zip archive? Will you end up with an archive with mixed compression modes.
Another possibility would be that when reading a zip file I check if it uses at least one zstd file and switch internally to bit7z....so the change happens invisble to the user.....but still you get the same questions as above
I will think a bit about it.

Little update already: Currently I don't see an option to specify zstd for adding files in bit7z....asking the author now....


ah yeah...by the way...there were some changes lately so here's the latest wip:

UI:
- fixed: remove datasource XML file when hash folder changed (should only happen when XML file was internally created from an exe export)

Core:
- added: "available:" filter which limits the machine selection to files you have. You might notice a count difference in total vs filtered even when you got all files. This is based on the fact that a) empty sets are excluded and b) there are clones which   are totally included in their parent, so -in split mode- you don't have a standalone file/folder for it. So don't worry about the count.
- misc:  "file:" filter, in case you work with softwarelist collections, you'd need to prefix single entries with sl-name#pacman (e.g. a2600#pacman) to specify which set you're refering to. Not needed for single sl files or standard dats though.
- misc:  dupe output now shows all found paths belonging to a machine
- misc:  some earlier cancel returns in fix wrong named disk/sample/rom/machine
- fixed: dupes can be listed multiple times

8
clrmame Discussion / clrmame v0.2 released
« on: 20 January 2025, 19:59 »
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_v02.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:
- added: #thread selector
- added: scanner, Clipboard->Copy Machine Names option
- fixed: rebuilder log Copy To Clipboard misses linefeed

Core:
- added: you can define a thread pool size which is used for parallel scanning rebuilding/etc. Either you keep auto (default value) or pick the number of threads used for the pool yourself. "auto" picks a high value when you work with compressed files and a low value when using decompressed ones. A low value is better when you run into file seek overhead issues. If you're not happy with auto, you can try to tweak it yourself.
- fixed: scanner and rebuilder ignore output path, mergemode, pattern changes when xml data hasn't changed
- misc: more effective reimplementation of thread pools and thread queueing
- misc: prefetching sha1s on an archive if needed (speed increase, espically for archives with lots of files)
- misc: rebuilder: major update / reworked internal logic, less complex, way smaller memory footprint especially for dats out of hell (dats with thousands of dupe files), faster.
- misc: scanner, smaller memory footprint and speed update for "dats out of hell"
- fixed: scanner, accidently getting a crc32 on sub folders showing an error
- fixed: scanner, detection of unnneeded empty sub folders failed
- fixed: scanner, using old archive name when trying to remove some unneeded files causing can't remove files/can't access messages
- misc:  updated bit7z to 4.0.9
- misc:  updated pugixml to 1.15
- misc:  use "-" instead of "_" for replacing illegal file chars (cmpro align)
- misc:  monitoring hash folder for -listsoftware exports. In case of a change, cached data is ignored and a new export is generated. This is more for users which update their hash folders during the development phase of MAME on a regular basis and don't want to recompile the binary.
- misc: use one version number, fixed a tooltip typo ;-)

9
clrmame Discussion / Happy New Year...
« on: 12 January 2025, 14:52 »
Well well well....first of all...Happy New Year...2025.....a little bit late but anyhow, I'm alive and working...

A new version is somewhere around the corner.....I've spent the last days on doing some major rework on the rebuilder. Why? Well, it really performed lousy for "dats out of hell"....
For example, the progetto icons datfile which has 34000 icons where roughly 5000 are dupes....but running such a colleciton through the rebuilder means that it finds ~5000 matches for each of the ~5000 files....the rebuilder in clrmame wasn't well designed for that....and the memory footprint went up very high....and of course the rebuilding time.

The good thing is: The new implementation is way better. Plus it's also faster than the old one.
Just to give an impression: old cmpro rebuilder on the icons: 1m57s
clrmame rebuilder (while being fine on other stuff it totally failed here): not worth mentioning....I quit after an hour ;-)
reworked clrmame rebuilder for the next version: 1m42s

I have some minor stuff on my list before doing a release...but here's already a little sneak peak:



UI:
- added #thread selector

Core:
- misc: major rebuilder update / reworked internal logic, less complex, way  smaller memory footprint especially for dats out of hell (dats with thousands of dupes), faster.
 
- misc: reimplementation of thread pools
 
- added: you can define a thread pool size which is used for parallel scanning rebuilding/etc. Either you keep auto (default value) or pick the number of threads used for the pool yourself. "auto" picks a high value when you work with compressed files and a low value when using decompressed ones. A low value is better when you run into file seek overhead issues. If you're not happy with auto, you can try to tweak it yourself.

- fixed: scanner and rebuilder ignore output path, mergemode, pattern changes when xml data hasn't changed
- fixed: scanner, accidently getting a crc32 on sub folders showing an error
- fixed: scanner, detection of unnneeded empty sub folders failed
- fixed: scanner, using old archive name when trying to remove some unneeded files causing can't remove files/can't access messages
- misc:  updated bit7z to 4.0.9
- misc:  use "-" instead of "_" for replacing illegal file chars (cmpro align)
- misc:  monitoring hash folder for -listsoftware exports. In case of a change, cached data is ignored and a new export is generated. This is more for users which update their hash folders during the development phase of MAME on a regular basis and don't want to recompile the binary.
- misc: use one version number ;-)


10
Sorry, no profiler news at the moment.
Oddi's remark about slowlyness when scanning progetto snaps keeps me a bit busy these days. Not only that I already found some hash lookups which can be optimized a bit, I wonder how threading can influence the scanning speed.
Now John IV would say: I'm getting 1 second for a full scan, so threading rocks. yes, it does...when you're on a sdd...when you're on a hdd, too many threads may lead to too many seek operations which may slow the process down.
In case of an unpacked progetto collection (we talk about > 300000 files), each file's crc32 needs to be calculated....which is a lot seeking, reading and calculating (even if the actual calc is quickly done). Using too many threads may slow down the process....so currently I will do some benchmarking what the best value is.....I keep you updated with the output. Even if I see that too many threads slow it down, I somehow need a way to determine a good value (ok, I can show an input box where you simply define it as last possible solution ;-))

11
clrmame Discussion / a nightly cmpro...
« on: 07 December 2024, 20:06 »
if anyone still use the old tool....

https://mamedev.emulab.it/clrmamepro/binaries/cmpro64_20241207.zip
(64bit exe only)

it only fixes a problem with not detecting unneeded files in a sample parent set..... The bug was in there since decades I think ;)

Definetly time to use the new tools ;-)

12
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_008_015_1.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:
misc: restore default window positions when positions get out of range
misc: increase display time of tooltips
misc: slightly regrouped ui elements
added: scanner, context menu option to restore old scan results on startup and selection

Core:
misc: updated spdlog to 1.15.0
misc: updated to 7zip 24.09
misc: don't reload xml and build up structures when they are already in memory and need no refresh
misc: align and share rebuilder/scanner common routines
added: scanner, samples, supporting flac
fixed: scanner, typo in wine script
fixed: scanner, internal sha1 check for files with identical crc32 but different sha1 values isn't run
fixed: scanner, samples, wrong named files (case check only) aren't fixed
fixed: scanner, samples, wrong named files (case check only) also appear as unneeded
fixed: scanner, samples, decompressed files are marked as unneeded
fixed: scanner, samples, machines which reference themselves via sampleOf are marked as unneeded in full mode
fixed: scanner, confusing 'can't remove/backup' messages in case of circular renames between different sets
fixed: scanner, confusing 'can't remove/backup' messages in case a wrong file with a right name is replaced with a fill in

13
clrmame Discussion / Sneak....
« on: 25 November 2024, 16:45 »
Well, this week a new MAME will most likely be released....so I will wait for that one to see if something weird happens or not....This month I was pretty busy with real life but I found a little time to do some changes. Mainly I was working on some code moves here and there, simply because rebuilder and scanner have a lot in common...so it was clear to make such functions commonly used instead of having them doubled (which was still a bit of a relict when they weren't merged). I also had a look at sample scanning which had some issues

Core:
- misc:  updated spdlog to 1.15.0
- misc:  updated to 7zip 24.08
- misc:  don't reload xml and build up structures when they are already in memory and need no refresh
- misc:  align and share rebuilder/scanner common routines
- fixed: scanner, internal sha1 check for files with identical crc32 but different sha1 values isn't run
- fixed: scanner, samples, wrong named files (case check only) aren't fixed
- fixed: scanner, samples, wrong named files (case check only) also appear as unneeded
- fixed: scanner, samples, decompressed files are marked as unneeded
- fixed: scanner, samples, machines which reference themselves via sampleOf are marked as unneeded in full mode
- added: scanner, samples, supporting flac

UI:
- misc:  restore default window positions when positions get out of range
- misc:  increase display time of tooltips
- misc:  slightly regrouped ui elements
- fixed: scanner, typo in wine script

I also have some nice things on the request list...(...and surely there is the profiler topic....)...will see what I can do...
To name 3 of the request (but don't count on them soon)
- restore scan results on startup / change of xml
- only keep 1 file instead of fix dat and scan result since they nearly contain the same information
- maybe have some tree like output in the rebuilder to show which machines were rebuilt

14
https://mamedev.emulab.it/clrmamepro/binaries/clrmame_007_014.zip

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

UI:

misc: scanner, contextmenu option to sort by name instead of description
misc: Linux/Wine, users can get data from a (linux) mame binary. A wrapper script is included (requires a chmod +x ./wrapper.sh once though)
misc: Linux/Wine, minor changes regarding line breaks or layout

Core:

misc: in case of a -listsoftware data basis (either exe export or dat), additional sl hashes are added from either the used exe's hash folder (prio 1) or HashFolder specified in the settings xml file (prio 2).
fixed: some absolute paths aren't made 32k path length aware, this can lead to sideeffects, e.g. when doing backups (esp. in Linux/Wine)
fixed: scanner, removal of matched files from addPath/backup only works when at least one file was missing
fixed: scanner, removal of unneeded files stopped after first error during a delete process so that other files in the queue were skipped
fixed: scanner, fixing wrong named files/folders which only differ by a character case change fails
fixed: scanner, unneeded sample files are detected but not reported or fixed
fixed: scanner, (rare) "can't read" error message when accidently testing a folder for being a chd
fixed: scanner, (rare) fixing wrong named files/folders which only differ by a character case but map to multiple machines keeps only 1 instance (e.g. 3DO->3do & 3dobios)
fixed: scanner, circular renames inside a machine throw rename errors and only get resolved in a second run

15
clrmame Discussion / Nightly.....
« on: 27 October 2024, 18:32 »
While playing around with the Linux MAME export I totally forgot to say what else is new in the nightly build:

https://mamedev.emulab.it/clrmamepro/binaries/clrmame_006_013_20241027.zip

UI:
- misc: scanner, contextmenu option to sort by name instead of description
- misc: Linux/Wine, users can get data from a (linux) mame binary. For this, a wrapper script is included. You need to do a chmod +x ./wrapper.sh once though
- misc: Linux/Wine, minor changes regaring line breaks or layout

Core:
- fixed: scanner, unneeded sample files are detected but not reported or fixed
- fixed: scanner, rare "can't read" error message when accidently testing a folder for being a chd
- misc:  in case of a -listsoftware data basis (either exe export or dat), additional sl hashes are added from either the
used exe's hash folder (prio 1) or HashFolder specified in the settings xml file (prio 2).

A big "thank you" goes out to Shoegazer for the linux testing and finally pointing me in the right direction to fix the (hopefully) last outstanding problem.

So what's next?
Well...the upcoming week will most likely bring a new MAME version. As usual, I will check the scanner/rebuilder core with it to see if something weird pops up. So expect a new 'official' release after the MAME release.

After that I hope I finally find time to start with the profiler....that's the short view...on the long view I was also thinking about a batcher (currently I wonder if there is something which a profiler can already do, i.e. loading and scanning multiple dats, but I also have some ideas for a batcher which would act like a scheduler where you add your scan/rebuild jobs in a queue like thing....time will tell...profiler comes first).
And yes....I'm still sure to release the source :-)

16
UI:
- misc:  SHIFT clicking the clear buttons clear the full combo box
- misc:  when context menu show info->size is enabled, also show the sum on software list level
- misc:  remember last used tab on close/start
- fixed: restore window positions incl. vertical break isn't always working

Core:
- misc:  set current folder back to application folder after scan/rebuild to avoid folder locking
- fixed: scanner, already moved "missing but fixable" unpacked files removes the moved file afterwards when they were also marked as unneeded in their source position
- fixed: scanner, a failed file operation on "missing but fixable" removes the file when they were also marked as unneeded in their source position
- fixed: obsolete error message when using file filters
- fixed: avoid 'can't read' ios_base::failbit set errors on files where the file length is smaller than the minimal buffer for the operation (e.g. id or header read)
- fixed: scanner, obsolete "can't access" messages during fix unneeded due to not correctly filtering already removed archives


https://mamedev.emulab.it/clrmamepro/binaries/clrmame_006_013.zip

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

17
clrmame Discussion / MAME .270 ...and the new tools
« on: 25 September 2024, 18:14 »
ok...a quick look at the new MAME might give you a "can't access scherrym" error during fixing....but that's only temporary...and seems to get resolved automatically...however, I will have a look at the source of this message tomorrow..until now: nothing critical....

18
Combined rebuilder/scanner in one commandline and one UI version, let's name it clrmame for now

Core:

misc: scanner, remove matches from AddPath option now removes ALL matches, no matter if they were just real fill-ins for a missing file or already existing in a rompath
misc: hiding sample specific problems while loading datfile in rebuilder and in scanner when no sample paths are set
misc: scanner, skip not accessible unneeded files (loglevel trace can show information about them)
fixed: using relative paths fails after first scan due to wrong current folder
fixed: when an elapsed time takes less than 0 seconds, no value as elapsed time is shown ;-)
fixed: hash calculation of empty files
fixed: scanner, some unneeded files won't be removed but aren't reported either when fix is enabled

UI:

added: scanner, context menu option to show clone names (in full merge mode only) which have issues right beside the set name up to 5, if there are more, you can see all in tooltips
misc: merged rebuilder and scanner, switchable via tabs, can run in parallel (unless path sharing issues are detected)



https://mamedev.emulab.it/clrmamepro/binaries/clrmame_005_012.zip

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

19
clrmame Discussion / Sneak Peak
« on: 11 September 2024, 19:01 »
Well...pretty quiet here recently?

Well, that doesn't mean nothing happens:


UI:

- added: scanner, context menu option to show clone names (in full merge mode only) which have issues right beside the set name
         up to 5, if there are more, you can see all in tooltips

Core:


- fixed: using relative paths fails after first scan due to wrong current folder
- fixed: when an elapsed time takes less than 0 seconds, no value as elapsed time is shown ;-)
- fixed: hash calculation of empty files
- fixed: scanner, some unneeded files won't be removed but aren't reported either when fix is enabled
- misc:  scanner, remove matches from AddPath option now removes ALL matches, no matter if they were just real fill-ins for a missing file or already existing in a rompath
- misc:  hiding sample specific problems while loading datfile in rebuilder and in scanner when no sample paths are set



ah well..that's not all.....there is an additional picture attached....

20
clrmame Discussion / scanner 0.04, rebuilder 0.11
« on: 30 August 2024, 07:17 »
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

https://mamedev.emulab.it/clrmamepro/binaries/scanner004_rebuilder011.zip


Core:

- added: scanner, option to remove fill-in files from addpaths (this can include backup when 'act as' option is on). Only files which were actually needed in the rompaths are removed
- misc:  changed elapsed time format and show an overall duration
- fixed: handling of loadflag="continue" fails in software list based xmls
- fixed: scanner, fixing missing roms/disks can accidently move the rom/disk when it shouldn't
- fixed: scanner, parentfolders of fill-in files can be touched (timestamp update) when they shouldn't
- fixed: scanner, fix missing roms/disks can accidently look in a folder where the match is not present leading to can't access/can't backup messages

UI:

- added: checkbox for upper mentioned remove fill-in files option
- misc: updated some flyovers




By the way: If you have a 0 byte gekimaka.zip in your rompath, you need to remove it manually first. That could have been created with the .03 scanner.....

Pages: [1] 2 3 4 5 6 ... 12

Page created in 0.102 seconds with 18 queries.