Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - Roman

Pages: [1] 2 3 4 5 6 ... 11

clrmame Discussion / Sneak Peak

« on: 11 September 2024, 19:01 »

Well...pretty quiet here recently?

Well, that doesn't mean nothing happens:

UI:

- added: scanner, context menu option to show clone names (in full merge mode only) which have issues right beside the set name
up to 5, if there are more, you can see all in tooltips

Core:

- fixed: using relative paths fails after first scan due to wrong current folder
- fixed: when an elapsed time takes less than 0 seconds, no value as elapsed time is shown ;-)
- fixed: hash calculation of empty files
- fixed: scanner, some unneeded files won't be removed but aren't reported either when fix is enabled
- misc: scanner, remove matches from AddPath option now removes ALL matches, no matter if they were just real fill-ins for a missing file or already existing in a rompath
- misc: hiding sample specific problems while loading datfile in rebuilder and in scanner when no sample paths are set

ah well..that's not all.....there is an additional picture attached....

clrmame Discussion / scanner 0.04, rebuilder 0.11

« on: 30 August 2024, 07:17 »

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

https://mamedev.emulab.it/clrmamepro/binaries/scanner004_rebuilder011.zip

Core:

- added: scanner, option to remove fill-in files from addpaths (this can include backup when 'act as' option is on). Only files which were actually needed in the rompaths are removed
- misc: changed elapsed time format and show an overall duration
- fixed: handling of loadflag="continue" fails in software list based xmls
- fixed: scanner, fixing missing roms/disks can accidently move the rom/disk when it shouldn't
- fixed: scanner, parentfolders of fill-in files can be touched (timestamp update) when they shouldn't
- fixed: scanner, fix missing roms/disks can accidently look in a folder where the match is not present leading to can't access/can't backup messages

UI:

- added: checkbox for upper mentioned remove fill-in files option
- misc: updated some flyovers

By the way: If you have a 0 byte gekimaka.zip in your rompath, you need to remove it manually first. That could have been created with the .03 scanner.....

clrmame Discussion / new scanner and MAME .269

« on: 29 August 2024, 06:49 »

End of month, a new MAME is out, a fantastic opportunity to find issues in the new tools...

a quick look at it shows me that there can be some "can't access" and "can't backup" messages around wontame, dokodemo, gekimaka sets.
Such sets have roms and chds and the missing files in the MAME.269 cycle can be taken from other sets (i.e. pingu). Unfortunately the new scanner accidently also takes the chd folder of pingu for fixing the roms...where it doesn't find the roms and reports the issues.

Working on a fix (choosing the right source (dir or archive) in case of multiple sources) now and I guess a new version will be out soon.

clrmame Discussion / scanner 0.03, rebuilder 0.10, .....and some cmpro update released

« on: 22 August 2024, 16:16 »

https://mamedev.emulab.it/clrmamepro/binaries/readme.html

https://mamedev.emulab.it/clrmamepro/binaries/scanner003_rebuilder010.zip

Core:

- added: long path/filename support (32k)

- misc: update to bit7z 4.0.8

- misc: scanner: propagate changed file timestamp up to rom/samplepath level
- misc: scanner: improved preferedPackMethod a bit
- misc: scanner: whenever possible, move (instead of copy/delete) missing but fixable chds or decompressed roms

- fixed: erroneously allow \ as an machine name character

- fixed: scanner, stopping the scan during fix operations doesn't have an effect
- fixed: scanner, freeze when fixing unneeded decompressed files (or chds) (0.02.1)
- fixed: scanner, wrong named files which are also fill-ins for missing files can trigger an obsolete file removal operation (sdiamond MAME.268)
- fixed: scanner, unneeded files/folders contain multiple identical entries (aa3020 MAME.268)
- fixed: scanner, fixing a wrong named folder failed when additional pattern folders were involved (copy error)
- fixed: scanner, fixing a wrong named folder failed when folder can have multiple new names (aa3020/a3010/aa5000 MAME.268) (move error)
- fixed: scanner, prefered path lookup for software list collections never returned additional pattern folders (copy error)
- fixed: scanner, prefered path lookup for wrong named machines which only consist of chds failed (freeze during backup)
- fixed: scanner, removal of unneeded files can result in an empty folder which isn't removed

UI:

- added: scanner, always have software list collection sublevels (before a complete sl list only showed the complete top level)
- added: scanner, context menu option to auto expand scan results tree
- added: scanner, context menu list options are now also available on software list level

- fixed: active paths are not always remembered correctly when switching xmls or on initial load

- fixed: scanner, empty sofware lists show complete instead of empty icon
- fixed: scanner, scan results tree/log split isn't refreshed after minimizing/maximizing window

and in case you're interested in old cmpro:

4.049

fixed: falsely remove backup's _unknown folder in case of a failed backup which might remove already backuped and removed unneeded files. Only if _unknown folder isn't present before.
fixed: skip a non software list rompath if it is also assigned to a non active software list which would list all sets in that path as fully missing (combined mode only)
fixed: missing disk merging information within softwarelists, so that some identical chds stay in the clone set (combined mode only)
added: verify delta chds (via chdman)

clrmame Discussion / New scanner 0.02.1 out

« on: 15 July 2024, 19:43 »

UI:

- added: scanner log *and* tree can be X/Y resized now (mouse click+move between the 2 controls)
- added: clear button to clear all combo boxes for current 'profile' (rebuilder and scanner)
- added: clear button to remove not existing XML/EXE entries
- misc: increased the number of remembered xml entries to 10 (rebuilder and scanner)
- added: controls for unnneeded masks, include backup as addpath
- misc: improved storing of per-xml data
- fixed: filtermode setting isn't restored

Core:

- added: support more XML based datfiles
- added: unneeded masks (regular expression to filter out specifc unneeded files/folders) (e.g. ^.*\\(CHD|SL)$)
- added: handle backup as addpath option
- added: filter option supports prefix 'file:' to specify a file holding setnames used for filtering
- fixed: several obsolete "can't remove" errors based on not taking an in between rename into account
- fixed: machine/rom/disk/sample names may contain illegal path characters (incl. ending with .). Remove characters and warn about it
- fixed: prevent rom/disk with identical names but different hashes to get overwritten in decompressed standalone or full merged mode (e.g. sgi_mips chds). Filename gets prefixed with hash. MAME might fail loading it but at least the file is not lost
- fixed: fix-missing can skip one file from a batch and doesn't fix or report it
- fixed: application quits when directory scans cause an exception (e.g. access right denied)

ReadMe

Download

Feel free to support me

Update: 0.02.1 fixes a freeze when fixing unneeded decompressed files....

clrmame Discussion / Forum Spam Bot Protection

« on: 07 July 2024, 19:47 »

Looks like the forum reacts a bit weirdly and more strictly to posts today....I also had problems getting a reply through....used a different browser and it worked though.
Already informed the forum owner about it...maybe some software update causes this...

Sometimes logging out and in again helps, too.....

clrmame Discussion / MAME .267 and the new scanner

« on: 30 June 2024, 12:19 »

So new MAME is finally out....time to check the scanner on it... and yes, there are some things I need to look into.

Trivia (Questions Series 7, alt question ROM) [folder: gtsers7a - parent: gtsers1]
Here either cmpro or the new scanner complains about the name. The rom name ends with a "." which is not possible under Windows...cmpro removes it from the name, the new scanner seems to keep it, so you end in a rename loop if you scan with cmpro then with the new scanner and so on...I will fix that

cmpro also complains about some missing clone roms in cmv4zga, gtsers7a ,mspacmane , mspacmane2, pleiadsgmp and pacmanmr while the new scanner doesn't do that in its first run. So you may have the feeling everything is ok, while a 2nd scan reveals the "fixable but missing" ones. They also get fixed, but they should get fixed in the first run already OR it should be listed.

I will definetly check on those and what the reason for them is....so, if you find other things besides of these sets, let me know

clrmame Discussion / New scanner versus old cmpro

« on: 24 June 2024, 16:28 »

Trying to collect stuff here which may be helpful when you switch or play around with the new tools:

Differences in Merge Modes:
- split: this should be fully identical to cmpro
- standalone (new) versus not or unmerged (old): The new standalone mode is more accurate in my opinion since it includes all dependency files (parent/bios/devices) into the set. The old mode only takes over the parent ones (and with some tricks the bios)
- full merged mode: the full merge mode acts like cmpro's full merge mode with "full merge mode -> hash collision name" (this is not the default options) enabled. The naming there is a bit misleading. It simply means that all clone sets inside a packed parent set are stored in subfolders.

Regarding scan results there can be slight differences:
- the tree is finally sorted alphabetically be machine description, cmpro is by setname which leads to some weird output order
- some results might slighly differ since the new scanner is simply different and the categorization of a wrong named machine or a wrong placed machine or an uneeded machine differs a bit. But after fixing, the resulting sets should be the same.

Scanning software lists or using system default paths
There are no system default paths anymore and you don't need to go through rompath setup hell when it comes to software lists collections.
Simply us 1 rompath and use either one of the patterns (e.g. #BIOSSPLIT# which is identical to system default paths) or in case of scanning a -listsoftware collection, you can even keep that blank since it internally uses #SOFTLIST#. You only need to store your software lists in rompath subfolders named after the list though.
For example you use: rompath: c:\mame\softwarelist and inside there are folders like a2600, a2600_cass, c64_flop_misc, amiga_hardware and so on which hold the files for the belonging lists.

Parent/Clone relationships
In cmpro it it optional to use merge tags for roms/disks from the datfile to identify parent/clone relationships of rom files. The new applications fully rely on it (at least for non software lists).

Supported Datfile Formats
Old cmpro supports all kind of datfiles. New apps only support XML based ones which are more or less compatible to -listxml, -listsoftware or MAME's hashfile format. This doesn't mean you are limited to MAME.

clrmame Discussion / scanner 0.01 and rebuilder 0.08 released

« on: 20 June 2024, 06:52 »

Finally....

I've decided to put out the first instance of the new scanner (bundled with a rebuilder update).

If you followed the project here on the forum in the past, you'd know that this is not yet a full clrmamepro replacement. It's something I had fun with by rewriting things in a way clearer and more modern code style. If you want to know more about the benefits, you will find information in the readme file. There is no profiler yet (that's the next thing to do) but you are already able to switch between a handful of setups (e.g. a -listxml and -listsoftware profile) which actually should be enough for a MAME user. Regarding the UI versions, they are preliminary. Things might change a little or a lot. The goal of these releases is to get robust cores. I highly recommend to have a peak at the readme file and of course the usage (especially turning on fix options) is your very own risk.

If you're enjoying this little project as much as I do, feel free to support me

clrmame Discussion / New Scanner (WIP)

« on: 02 May 2024, 17:31 »

Well, it was pretty quiet the last weeks...so before you might think that nothing is happening, here is a little peak.
While I was indeed pretty busy with real life I can say that the scanner is still alive, so let's write a bit about it.

As you might know (https://www.emulab.it/forum/index.php?topic=9199.0), the scanner does already a pretty good job in scanning uncompressed and compressed sets, no matter how you organized them (plain, in multiple rompaths, in pattern styled subpaths (e.g. by year/manufacturer/etc) and scanning MAME -listxml output (or anything which followes this format), -listsoftware collections and single MAME hash files. It can do a 'new scan' or a 'scan' where the latter takes scan results information from a previous scan into account (similar to current cmpro).

And now, it also fixes all found problems.

- It is able to auto detect possible missing but fixable files (roms/chds) in your rompaths and in your backup path and everything in one (1) scan and it tries to find a best place for it (as long as you're using one rompath you wouldn't care...but there are people who use pattern-like storage or split chds and roms, etc....)
- It detects wrong names (machines/roms/chds) and fixes them of course
- It detects unneeded files wherever they are and puts them to the backup path before getting rid of them. It does not automatically zip them as cmpro does (see a thread on this forum about it) but takes over the format from the origin and reproduces the rom path structure inside the backup folder. If a file with the same name but different hash already exist there, the scanner will take care of it.

Attached are some screenshots:
- one showing the context menus which you can use to modify the output or copy data to the clipboard
- one showing an example output of scanning a software list collection (no need for multiple profiles or a complex setup...just scan)
- one showing an example output of scanning multiple rompaths and showing complete and partly missing but not fully missing ones...

So..what's missing for a 0.01 version

- doing some more tests with some different scenarios
- a handful of minor nice to haves (still not wanting to add all kind of requests though)
- I'm not yet fully satisfied with writing the scanresults file and fix dat file..it works fine but I think I can speed it up a bit
- need to write a documentation (yikes)

So what's next then after the scanner runs fine?

- most likely people will complain about everything, find bugs and have tons of ideas/requests...so I will look into it
- next major step then would be to combine rebuilder and scanner and have some kind of profiler

clrmame Discussion / Happy 2024

« on: 04 January 2024, 20:55 »

Wish you a peaceful and healthy 2024!

New Scanner coming along nicely....(Screenshots attached)
However, during the holiday season I only did some little work here and there....now fully concentrating on the core fixing part.
(And yes, you can have multiple rom and sample paths :-) )

clrmame Discussion / rebuilder 0.07 released

« on: 28 November 2023, 18:18 »

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v007.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

- fix software list rom sizes determination (wasn't limited to loadflag value)
- fix software list merging (SL/SL collections don't use merge attributes, so lookup by hash in a parent/clone relationship)
- don't use # in default pattern (rompath) names since such names would be cut off when used in mame.ini due to comment handling
- pattern names can't end with '.' (Windows doesn't like this), replaced cases with "_"
- minor changes to the stats count output
- updated 3rd party libs (spdlog, bit7z, pugixml)

clrmame Discussion / clrmamepro 4.048d released

« on: 28 November 2023, 18:16 »

4.048d
fixed: softwarelists can lack some possible mergings
fixed: possible assertion during delta checks

clrmame Discussion / Some new scanner wip

« on: 22 November 2023, 19:02 »

While wasting a little bit of time with delta chd support in old cmpro, I recently found a little time to work on the new scanner again..

So what's the status?...well, it scans nicely:

- mame -listxml output, mame hash files, mame -listsoftware output
- full/split/standalone merged
- multi rom/sample paths suppport
- pattern scanning supported (if you keep your sets split by bios etc)
- roms, disks and samples, compressed (7z/zip) or uncompressed
- writing of fix dats
- writing of report output
- remembering old scan information for a faster scan
- commandline available (compareable to new rebuilder)
- preliminary gui available (compareable to new rebuilder)

What's on the to do list:

Well, the actual fixing part, but that be rather simple since it simple needs to run through a list of tasks like
'rename this to that', 'delete this file here', 'add file from there to there'...

Got a small wishlist already some some UI related features here and there...but they don't have any priority...

and of course.....after that, a profiler...

Ever wanted to scan a full software lists collection? Remember the pain in cmpro to set it up? Well, now it's just adding a rompath and hitting start...a little screenshot shown below should give you an idea. Generally it should be way easier to set it up...no need to define 'system default paths', no need to setup thousands of distinct rompaths just for softwarelists.....

clrmame Discussion / 4.4048c

« on: 04 November 2023, 17:20 »

4.048c
fixed: wrong "Error while..." message after delta operation succeeded but backup is disabled.
misc: minor changes to the delta output messages

Keep in mind: https://github.com/mamedev/mame/issues/11675
....when this is fixed I will check if the V5 restriction can be taken back and some more chds can be converted....the github issue is about a V5 one though....

clrmame Discussion / cmpro 4.048b

« on: 29 October 2023, 18:44 »

4.048b
fixed: wrong delta chd proposals in full merged mode (identical chds)

...and yes, it seems that converting the susume chd to a delta fails (also with manual chdman usage), JohnIV already filed a bug report to the MAME team....

clrmame Discussion / cmpro 4.048a

« on: 29 October 2023, 12:12 »

4.048a
fixed: unneeded scan crashes/exits
misc: limiting delta chd support for chd version >=5

clrmame Discussion / clrmamepro 4.048 released

« on: 28 October 2023, 18:49 »

4.048
added: delta chd support, show warnings, support convert/revert of chds
fixed: MAME 260 "samples" issue (is shown as unneeded and not found)

For delta chd support, see here:
https://www.emulab.it/forum/index.php?topic=9157.msg25906#msg25906

clrmame Discussion / delta chds, cmpro test version

« on: 26 October 2023, 18:05 »

Ok, here we go....a test version for some delta chd features.

Disclaimer: This is not an official release, it's just a 64bit cmpro exe file which can replace yours. Do it on your own risk. Any chd convert/revert operation, you do it on your own risk.

What are the delta chd features (screenshots here: https://www.emulab.it/forum/index.php?topic=9148.msg25902#msg25902) ?

In Settings->Compressor...->CHDMan, you find a new section "Delta CHDs". It also holds two new command line boxes for converting and reverting. They are filled with the default copy operations, don't change them.
The section holds options where you can decide if cmpro
- should warn you about chds which can be converted to deltas
- chds which are deltas
- or you don't want to see any warnings at all (default).

There is also a checkbox (ticked by default) which enables backup when a convert/revert operation actually takes place.

After running a (new) scan and having a warning option enabled, you might see warnings in the warnings window below the scanner. It can list something like:

- Non-delta chd: some.chd Convertable via someother.chd ( showed warning about non-delta chds )

- Delta chd: some.chd Revertable via someother.chd (showed warning about delta chds)

- Bad parent delta chd: some.chd Revertable via someother.chd (error: parents can't be deltas)

- Bad clone delta chd: some.chd Revertable via someother.chd (error: the delta clone uses a different chd than the current parent)

- Can't convert/revert some.chd. Can't find a non delta CHD with SHA1: somesha1 (error: parent is not available)

If one or more of the messages are shown, you can use the popup contextmenu on one of the lines (you can also select multiple) and select one of the delta chd options in the menu. You can convert or revert all/selected. Depending on the selection you see either the one or the other or both options...

When selecting convert or revert, chdman is called and run in the background. If something went wrong or your used ctrl-c to exit chdman and you got multiple files in the queue, you will be asked if you want to continue.
If you ticked the backup option (you should), delta and absolute chds are copied to your backup folder. For this, subfolders are created in there _chd_abs_ and chd_delta_.

Known things:

- delta chds are always smaller? No, there are cases, especially when it comes to examples where the parent chd is a v4 baddump and the clone is a v5 chd, the delta version is slightly bigger

- when a chd is converted and reverted, I get the original file back? No, not necessarily. There are chds which were created with an older version of chdman and converting/reverting them with the current one ends up with a slightly smaller file. This is most likely based on the fact that the newer chdman uses a newer version of lzma which is responsible for the size loss. However, the raw data and its sha1 will be the same.

- when thinking about reverting, cmpro's unneeded/set check helps to find parent chds which might be located elsewhere or named incorrectly

- Can I convert all clone chds? If there is a belonging parent chd, yes (when CHD version is >= 3). Belonging? Well, there are sets which have more than one chd. To identify a belonging chd, the chd's "region" (and for software lists, the "part") information is used for a match. If a chd is not listed with a region/part, cmpro won't use it for any delta operations.

Well..happy testing...and don't forget: it's your fault when something goes wrong ;-)

Don't know if it's worth all the trouble....in times of multi terabyte HDs.....but the warnings might be useful.

clrmame Discussion / clrmamepro 4.047b released

« on: 22 October 2023, 16:58 »

4.047b
fixed: don't mark delta chds as missing
misc: updated unrar (6.24.0)

I will think about additional options (like optionally marking found not-delta chds and maybe a conversion tool/update opption) but for this I need more time.