EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - Roman

Pages: [1] 2 3 4 5 6 ... 11
1
clrmame Discussion / New Scanner (WIP)
« on: 02 May 2024, 17:31 »
Well, it was pretty quiet the last weeks...so before you might think that nothing is happening, here is a little peak.
While I was indeed pretty busy with real life I can say that the scanner is still alive, so let's write a bit about it.

As you might know (https://www.emulab.it/forum/index.php?topic=9199.0), the scanner does already a pretty good job in scanning uncompressed and compressed sets, no matter how you organized them (plain, in multiple rompaths, in pattern styled subpaths (e.g. by year/manufacturer/etc) and scanning MAME -listxml output (or anything which followes this format), -listsoftware collections and single MAME hash files. It can do a 'new scan' or a 'scan' where the latter takes scan results information from a previous scan into account (similar to current cmpro).

And now, it also fixes all found problems.

- It is able to auto detect possible missing but fixable files (roms/chds) in your rompaths and in your backup path and everything in one (1) scan and it tries to find a best place for it (as long as you're using one rompath you wouldn't care...but there are people who use pattern-like storage or split chds and roms, etc....)
- It detects wrong names (machines/roms/chds) and fixes them of course
- It detects unneeded files wherever they are and puts them to the backup path before getting rid of them. It does not automatically zip them as cmpro does (see a thread on this forum about it) but takes over the format from the origin and reproduces the rom path structure inside the backup folder. If a file with the same name but different hash already exist there, the scanner will take care of it.


Attached are some screenshots:
- one showing the context menus which you can use to modify the output or copy data to the clipboard
- one showing an example output of scanning a software list collection (no need for multiple profiles or a complex setup...just scan)
- one showing an example output of scanning multiple rompaths and showing complete and partly missing but not fully missing ones...


So..what's missing for a 0.01 version

- doing some more tests with some different scenarios
- a handful of minor nice to haves (still not wanting to add all kind of requests though)
- I'm not yet fully satisfied with writing the scanresults file and fix dat file..it works fine but I think I can speed it up a bit
- need to write a documentation (yikes)


So what's next then after the scanner runs fine?

- most likely people will complain about everything, find bugs and have tons of ideas/requests...so I will look into it
- next major step then would be to combine rebuilder and scanner and have some kind of profiler


2
clrmame Discussion / Happy 2024
« on: 04 January 2024, 20:55 »
Wish you a peaceful and healthy 2024!

New Scanner coming along nicely....(Screenshots attached)
However, during the holiday season I only did some little work here and there....now fully concentrating on the core fixing part.
(And yes, you can have multiple rom and sample paths :-) )

3
clrmame Discussion / rebuilder 0.07 released
« on: 28 November 2023, 18:18 »
https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v007.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html


- fix software list rom sizes determination (wasn't limited to loadflag value)
- fix software list merging (SL/SL collections don't use merge attributes, so lookup by hash in a parent/clone relationship)
- don't use # in default pattern (rompath) names since such names would be cut off when used in mame.ini due to comment handling
- pattern names can't end with '.' (Windows doesn't like this), replaced cases with "_"
- minor changes to the stats count output
- updated 3rd party libs (spdlog, bit7z, pugixml)

4
clrmame Discussion / clrmamepro 4.048d released
« on: 28 November 2023, 18:16 »
4.048d
fixed: softwarelists can lack some possible mergings
fixed: possible assertion during delta checks


5
clrmame Discussion / Some new scanner wip
« on: 22 November 2023, 19:02 »
While wasting a little bit of time with delta chd support in old cmpro, I recently found a little time to work on the new scanner again..

So what's the status?...well, it scans nicely:

- mame -listxml output, mame hash files, mame -listsoftware output
- full/split/standalone merged
- multi rom/sample paths suppport
- pattern scanning supported (if you keep your sets split by bios etc)
- roms, disks and samples, compressed (7z/zip) or uncompressed
- writing of fix dats
- writing of report output
- remembering old scan information for a faster scan
- commandline available (compareable to new rebuilder)
- preliminary gui available (compareable to new rebuilder)

What's on the to do list:

Well, the actual fixing part, but that be rather simple since it simple needs to run through a list of tasks like
'rename this to that', 'delete this file here', 'add file from there to there'...

Got a small wishlist already some some UI related features here and there...but they don't have any priority...

and of course.....after that, a profiler...

Ever wanted to scan a full software lists collection? Remember the pain in cmpro to set it up? Well, now it's just adding a rompath and hitting start...a little screenshot shown below should give you an idea. Generally it should be way easier to set it up...no need to define 'system default paths', no need to setup thousands of distinct rompaths just for softwarelists.....

6
clrmame Discussion / 4.4048c
« on: 04 November 2023, 17:20 »
4.048c
fixed: wrong "Error while..." message after delta operation succeeded but backup is disabled.
misc:  minor changes to the delta output messages



Keep in mind: https://github.com/mamedev/mame/issues/11675
....when this is fixed I will check if the V5 restriction can be taken back and some more chds can be converted....the github issue is about a V5 one though....

7
clrmame Discussion / cmpro 4.048b
« on: 29 October 2023, 18:44 »
4.048b
fixed: wrong delta chd proposals in full merged mode (identical chds)


...and yes, it seems that converting the susume chd to a delta fails (also with manual chdman usage), JohnIV already filed a bug report to the MAME team....

8
clrmame Discussion / cmpro 4.048a
« on: 29 October 2023, 12:12 »
4.048a
fixed: unneeded scan crashes/exits
misc: limiting delta chd support for chd version >=5

9
clrmame Discussion / clrmamepro 4.048 released
« on: 28 October 2023, 18:49 »
4.048
added: delta chd support, show warnings, support convert/revert of chds
fixed: MAME 260 "samples" issue (is shown as unneeded and not found)


For delta chd support, see here:
https://www.emulab.it/forum/index.php?topic=9157.msg25906#msg25906

10
clrmame Discussion / delta chds, cmpro test version
« on: 26 October 2023, 18:05 »
Ok, here we go....a test version for some delta chd features.

Disclaimer: This is not an official release, it's just a 64bit cmpro exe file which can replace yours. Do it on your own risk. Any chd convert/revert operation, you do it on your own risk.

What are the delta chd features (screenshots here: https://www.emulab.it/forum/index.php?topic=9148.msg25902#msg25902) ?

In Settings->Compressor...->CHDMan, you find a new section "Delta CHDs". It also holds two new command line boxes for converting and reverting. They are filled with the default copy operations, don't change them.
The section holds options where you can decide if cmpro
- should warn you about chds which can be converted to deltas
- chds which are deltas
- or you don't want to see any warnings at all (default).

There is also a checkbox (ticked by default) which enables backup when a convert/revert operation actually takes place.

After running a (new) scan and having a warning option enabled, you might see warnings in the warnings window below the scanner. It can list something like:

- Non-delta chd: some.chd Convertable via someother.chd ( showed warning about non-delta chds )

- Delta chd: some.chd Revertable via someother.chd (showed warning about delta chds)

- Bad parent delta chd: some.chd Revertable via someother.chd (error: parents can't be deltas)

- Bad clone delta chd: some.chd Revertable via someother.chd (error: the delta clone uses a different chd than the current parent)

- Can't convert/revert some.chd. Can't find a non delta CHD with SHA1: somesha1 (error: parent is not available)

If one or more of the messages are shown, you can use the popup contextmenu on one of the lines (you can also select multiple) and select one of the delta chd options in the menu. You can convert or revert all/selected. Depending on the selection you see either the one or the other or both options...

When selecting convert or revert, chdman is called and run in the background. If something went wrong or your used ctrl-c to exit chdman and you got multiple files in the queue, you will be asked if you want to continue.
If you ticked the backup option (you should), delta and absolute chds are copied to your backup folder. For this, subfolders are created in there _chd_abs_ and chd_delta_.

Known things:

- delta chds are always smaller? No, there are cases, especially when it comes to examples where the parent chd is a v4 baddump and the clone is a v5 chd, the delta version is slightly bigger

- when a chd is converted and reverted, I get the original file back? No, not necessarily. There are chds which were created with an older version of chdman and converting/reverting them with the current one ends up with a slightly smaller file. This is most likely based on the fact that the newer chdman uses a newer version of lzma which is responsible for the size loss. However, the raw data and its sha1 will be the same.

- when thinking about reverting, cmpro's unneeded/set check helps to find parent chds which might be located elsewhere or named incorrectly

- Can I convert all clone chds? If there is a belonging parent chd, yes (when CHD version is >= 3). Belonging? Well, there are sets which have more than one chd. To identify a belonging chd, the chd's "region" (and for software lists, the "part") information is used for a match. If a chd is not listed with a region/part, cmpro won't use it for any delta operations.


Well..happy testing...and don't forget: it's your fault when something goes wrong ;-)

Don't know if it's worth all the trouble....in times of multi terabyte HDs.....but the warnings might be useful.

11
clrmame Discussion / clrmamepro 4.047b released
« on: 22 October 2023, 16:58 »
4.047b
fixed: don't mark delta chds as missing
misc:  updated unrar (6.24.0)


 I will think about additional options (like optionally marking found not-delta chds and maybe a conversion tool/update opption) but for this I need more time.

12
While testing rebuilder/scanner the following drove me nuts...updating mame.ini with the generated paths from the upper mentioned methods didn't work. "MAME -rompath f:\mame\roms\#default pacman" did work, but adding "f:\mame\roms\#default" to mame.ini failed....so I wondered why...and oh man...feeling stupid......having a "#" as some prefix in a rompath is not a good idea for usage in an ini file. It simply takes the rest of the line as comment :-)

So your rebuilder settings.xml should use either your own values or the following new defaults (without the #)

      <TypeNames>
         <Bios>bios</Bios>
         <Mechanical>mechanical</Mechanical>
         <Default>default</Default>
         <Device>device</Device>
         <BiosSplit>bios_</BiosSplit>
      </TypeNames>


Will be updated for next release.....

13
clrmame Discussion / clrmamepro 4.047a released
« on: 26 September 2023, 15:22 »
4.047a
fixed: pick wrong file for backup during a failed rename where multiple files are involved


Tiny one...but might become useful for next MAME......

14
clrmame Discussion / Alive and kicking
« on: 12 September 2023, 11:57 »
It’s quiet here, isn't it?

One reason is that it's summertime and real life, holidays, Diablo4 and other stuff kept me busy.

But just wanted to say that I'm alive and kicking and working on a new scanner, similar to the new rebuilder.

Fortunately, the code basis of the rebuilder with the new concept of having views on XML data instead of implementing rules for different merge modes makes coding life very easy.

The scanner is in a state that it runs fine already in finding and reporting all issues. Didn't start the fixing part at all, but it is planned that this will be an easy set of straight forward archive or file operations like rename/delete/add.

Compared to the cmpro's scanner, where you can differ between missing, case, name, size, checksum etc checks, you will have all in one and it comes down to missing, wrong name or unneeded. You see, simplification is one of the goals. You won't need to specify anything to do this or that check.

One of the main things I wanted to find a good solution for are -taadaa- rompaths. As you might now not all people tend to put all sets into one (1) rompath. Some people split the collection by bios (so you got rompaths for e.g., neogeo, playchoice10 and so on), others may go even further and use something like a manufacturer/year combination. Since configurating rompaths is usually an annoying thing if #rompaths > 1, the new scanner will allow simply specifying a root folder plus some storing pattern expression, like the pattern the new rebuilder allows. If it comes to softwarelists (and loading a full software list xml), you can now simply use one rompath root. In cmpro it is really annoying to setup hundreds of paths. Of course you can also specify single ones, but maybe you get inspired to use some more structured paths.

Regarding speed I can say that it already looks promising compared to the cmpro scanner, some benchmarks will follow.
Keep in mind that with such large data as a MAME collection, you already have a fixed amount of time just to read in all the archives' central directories with your preferred zip/7z/file routine. You can't really get below that. Surely this time differs from SSD to HD and using/not using disk cache. Due to the new code basis, the on top part, the actual checking, becomes a very small part of the overall used time now.

For now, it is command line based but as you see from last rebuilder release, a simple UI can be added pretty quickly.
I'm currently also thinking about a better way for the command line output. Scanning a messy collection produces a lot of text output, so I think I may additionally write a result xml file which can be used for some plain html/xslt rendering purpose....time will tell.

It's still far away from a Profiler/Rebuilder/Scanner solution....but hey...let's see what the future brings...

So...just wanted to send out a small "hello world"....till next time.

Tada
Roman

15
clrmame Discussion / Rebuilder 0.06 released
« on: 04 May 2023, 18:35 »
Rebuilder 0.06 released

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v006.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

Doing source and destination matches now in multiple threads...gives a nice speed boost.

16
clrmame Discussion / rebuilder 0.05 released
« on: 14 April 2023, 20:06 »
Rebuilder 0.05 released

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v005.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

0.05 has a major unicode update, so now all your utf8 chars (no matter if in the datfile, in archives, in folders, in log output etc) should be handled correctly.

As a small extra...(I had a little time and wanted to see how easy it is to reuse the core) I've added rebuilderUI ;-)
It's a tiny UI for the rebuilder....the options should be easy to understand since they match the commandline parameters. The log list has a context menu and the window is resizable (things you don't see on the screenshot below).

RebuilderUI (rebuilderUI.exe) saves its settings in settings_ui.xml while the commandline rebuilder (rebuilder.exe) uses settings.xml. Both need the 7z.dll.

Hope you enjoy it...


17
clrmame Discussion / Rebuilder 0.04 released
« on: 12 March 2023, 17:58 »
2023-03-12 V0.04 released
- support reading of (split)rar/(not split)7z and writing of 7z files
- detection of zip, 7z, rar, chd files by byte signature (instead of extensions, but
  not within archives)
- selectable tempfolder in settings.xml
- minor speed up due to upfront matching size check
- updated various 3rd party libs, added 7z.dll
- ctrl-c will stop the rebuilding and cleans up temporary files/folders
- various internal cleanup

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v004.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

18
clrmame Discussion / clrmamepro 4.047 released
« on: 20 December 2022, 12:03 »
4.047
misc: updated ziparchive (4.6.9) and unrar (6.20.3)
misc: show warning when trimming removed a whitespace rom names with subfolders

19
clrmame Discussion / Rebuilder 0.03 released
« on: 05 October 2022, 15:49 »
2022-10-05 V0.03 released
- use a real move operation in case of copy/deleting single files (incl. chds)
- add option -u, --uselinks to generate filesystem hard or sym links instead doing a file copy or move operation

https://mamedev.emulab.it/clrmamepro/binaries/rebuilder_v003.zip
https://mamedev.emulab.it/clrmamepro/binaries/readme.html

20
clrmame Discussion / clrmamepro 4.046b released
« on: 18 August 2022, 14:01 »
4.046b
fixed: unsorted disk array can rarely cause random unneeded chd messages

Pages: [1] 2 3 4 5 6 ... 11

Page created in 0.127 seconds with 17 queries.

anything