Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Shoegazer]

Hi Roman, finally getting around to testing your shiny new clrmame tool and I have a few questions:

1. I currently use cmpro to scan batches of software lists from the profiler, by selecting multiple .xml software list files, and then cmpro handles them all at once.  I'm wondering how I would accomplish the same thing using clrmame?  It doesn't have to be "batch mode" per se, if it handles the scanning in a different way but achieves the same thing. 

2. cmpro permits rebuilds to the same target directory where the software is stored, without worrying about overwriting vital data.  I assume clrmame does the same thing, and this sort of action is safe?

3. There's no way to select the linux mame binary since clrmame filters out files that don't have .exe or .xml extensions (the linux mame binary is just "mame" with no extension).  Can you permit this in the future so the binary wouldn't have to be renamed for each clrmame operation?
 
Thanks!

[john iv]

Regarding item 1, I used to have 30 or so \hash\xmls that I would bring into the old clrmamepro but with the new one I just use the single file output of mame.exe -listsoftware and then use an XPath filter statement that only checks \software directories I collect.  This is much cleaner and less steps.

xp://softwarelist[@name='xegs' or @name='a800' or @name='a800_cass' or @name='a800_flop' or @name='a2600' or @name='a2600_cass' or @name='a5200' or @name='a7800' or @name='astrocde' or @name='coleco' or @name='coleco_homebrew' or @name='gameboy' or @name='gamegear' or @name='gba' or @name='gbcolor' or @name='intv' or @name='intvecs' or @name='jaguar' or @name='lynx' or @name='megadriv' or @name='microvision' or @name='n64' or @name='neogeo' or @name='nes' or @name='ngp' or @name='ngpc' or @name='pce' or @name='sms' or @name='snes' or @name='tg16' or @name='vboy' or @name='vectrex' or @name='videopac']/software

[Roman]

1) currently there is no profiler and no batcher (both planned though)....for software list use a -listsofware scan (either via exe or xml file). You only need to specify a rompath ( e.g. e:\mame\softwarelist) which holds the software list folders (e.g. e:\mame\softwarelist\a2600, e.g. e:\mame\softwarelist\nes, etc...). Screenshots were posted several times here on the forum already. If you're only interested in some sl results you can either use filters or disable the showing of fully missing software list findings via the context menu.

2) rebuilder does several checks on paths...destination can't be a subfolder of source, backup path, temp path etc...so...it's save :-) If you run a scan and rebuild at the same time (possible, but not really recommended), it also checks if paths collide

3) "since clrmame filters out files that don't have .exe or .xml " oh, good point, I guess I should allow an "all" setting. Surely you can simply enter the path in the edit box and don't use the browse button.

[Shoegazer]

Regarding item 1, I used to have 30 or so \hash\xmls that I would bring into the old clrmamepro but with the new one I just use the single file output of mame.exe -listsoftware and then use an XPath filter statement that only checks \software directories I collect.  This is much cleaner and less steps.

Thanks, John!  That's exactly what I did, and you're right, it's a better method overall.

Roman:

First, wow I'm impressed. You new tool not only looks better, but it's better organized, the options are clear and generally make more sense, and the speed... wow.  I've been using cmpro every month since your first release in 97, and if you had this tool back then, I'm pretty sure I would have had at least a year more of my life by now

So to your comments:

1. As I mentioned to John, the new method just works so well that I don't believe I'll need a batch profiler anymore (though it's cool that you're planning on it!). That said, would you consider an option to trigger mame -softwarelists within clrmame, so it doesn't need to be run out of band?

Also, I wrote a shell script to compare hash values between 0.269 and 0.270, iterating on each software set within the mame "hash" directory.  However, upon completion of the scan, clrmame did not pick up on changes for certain software sets that my shell script reported.  Can you help me understand why? The sets it missed are:

Code: [Select]

attache
coleco_homebrew
coleco
evio
lk3000
nes_kstudio
nes_ntbrom
pv1000
pv2000
smc777
supracan
Also, it reported on certain sets that were "unneeded" which is probably expected behavior since they don't seem to be in the mame -softwarelists output xml, however they DO have corresponding xml files in the hash folder, so I wonder - is this really an issue with mame -softwarelists not capturing all the sets, or are they somewhere else and I'm just blind?  Those sets are:

Code: [Select]

3do_m2
advantage
carbeena
gba_ereader
gtfore
ique
lisa2
pippin_flop
sat_vccart
snes_vkun
tandy6k
2. Great to know!  I'll just rebuild to the source folder then, since it's safe.  That's the way I was doing it in cmpro too, so I kindof expected it would be okay, but wanted to be sure before taking the leap.

3. After I typed my message I did also notice that the input box allows for free text, so yes you're right I can just add the path there.  That said, it's a little better to add an "all" setting. 

[john iv]

You can already call the -listxml and -listsoftware from the source box:
Just append the command to the .exe.  I'm 99% sure of the syntax, it's in the readme if not.

--
**Scanner:**

- XML / EXE - you need to enter a full filename of the datfile you want to use or
point to a MAME executable plus list its export function.
e.g. "F:\MAMEEmu\mame.exe -listxml" or "c:\temp\266.xml"

[Shoegazer]

Ah yes of course.  I suppose that should have been obvious; in cmpro I always thought of the path to the executable as without options to actually send flags to it, since I never used it for that purpose.  Good to know.  Thanks!

[Roman]

attache, coleco_homebrew, coleco, evio, lk3000, etc..

What do you mean exactly? -listsoftware lists them and clrmame complains about machines in there when something is missing in there (or unneeded)

regarding the 2nd list, yes, -listsoftware does not include all data from MAME's hashfolder. I don't know why (actually I also not asked MAME devs about it) but it's known for years. And yes, such entries will be reported as unneeded since they are not included in -listsoftware.
So a solution would be to keep them separated and scan them with a hash xml....
have to think about it more deeply....




Update: 1) or do you mean only the hash xml changed...well, if -listsoftware was used, clrmame stores an export of it in its export folder and rememebers the hash of the exe file....so pure changes in a mame hash folder aren't automatically detected. You could clean the exports folder though...

2) Hmm...maybe when doing a -listsoftware export from an exe file I could try to locate the hashfolder and match up if there are xml files in there which aren't included and simply merge them in......as I said...I need to think about it...but I like the idea.

[AntoPISA]

I've been wanting to add to my site for a while now a dat that includes the lists in \hash but not yet included in -listsoftware.

Maybe it's time...

Here https://mega.nz/file/4LBlQJBb#O9TZPHD4kY7yaU4EqxHasrBDW7PTqhdvXNvZ-ewNhIA a package with 2 xml:

- mame270-sl_full.xml All software lists in one file.
- mame270-slo.xml Only "orphan" lists not yet supported.

[Shoegazer]

attache, coleco_homebrew, coleco, evio, lk3000, etc..

What do you mean exactly? -listsoftware lists them and clrmame complains about machines in there when something is missing in there (or unneeded)
...
Update: 1) or do you mean only the hash xml changed...well, if -listsoftware was used, clrmame stores an export of it in its export folder and rememebers the hash of the exe file....so pure changes in a mame hash folder aren't automatically detected. You could clean the exports folder though...

Yes, I mean that in this particular case, my script compared files in the mame 0.269 and 0.270 hash directories, whereas clrmame compared 0.270 -listsoftware output with complete 0.269 sets located in the software rom path.  You can of course manually compare 0.269 with 0.270 versions of one of those software lists e.g. coleco and you'll see what I mean.  However as I now understand, we can't rely on the results of both methods to be the same because -listsoftware doesn't capture everything listed in the hash folder for some reason.

regarding the 2nd list, yes, -listsoftware does not include all data from MAME's hashfolder. I don't know why (actually I also not asked MAME devs about it) but it's known for years. And yes, such entries will be reported as unneeded since they are not included in -listsoftware.
So a solution would be to keep them separated and scan them with a hash xml....
have to think about it more deeply....
...
2) Hmm...maybe when doing a -listsoftware export from an exe file I could try to locate the hashfolder and match up if there are xml files in there which aren't included and simply merge them in......as I said...I need to think about it...but I like the idea.

I think this would be a suitable workaround (and you can use the two sets of software lists I noted above as a reference if it helps), and it would also help to catch future sync issues.  Another workaround might be to dynamically build a master software list file via the individual files in the hash directory, and just not use -listsoftware at all.  Yet another might be to expose an option to use the "cleaner" -listsoftware method or the "more complete" hash folder method (for whenever your batch feature is added).  In any event though, this is best addressed at the source, directly with the mame team.  It would be ideal if they could trap the condition prior to a given point release so it doesn't happen again.  Do you know whether a Github issue report or mametesters issue has been created for this?  Even if it's been known for years, without a formal report it might not be addressed.

I've been wanting to add to my site for a while now a dat that includes the lists in \hash but not yet included in -listsoftware.

Thanks! Yours is yet another decent workaround, and I use your helpful datfiles all the time, though in this particular case for the future it might be best to let Roman dynamically build a more inclusive list within clrmame itself so you don't have to do it manually for every release.

[Roman]

hmm...I've compared coleco.xml from 269 against 270 and there are only 3 changed descriptions. Surely clrmame will show the updated description if the file would be missing...but of course nothing really changes regarding the content of the files....

[Roman]

ok...my idea was pretty easy to add:

So in case of an exe based -listsoftware export, the tool now checks if there is a hashfolder for the exe file, takes all *.xml files in there which are not already matching a softwarelist name in the -listsoftware export and additionally loads them to the collection....
works fine...so the only thing which is assumed is that the hash xml filename does match its stored softwarelist name attribute value.....

So...no more overseen skipped software list hashes....(unless -listsoftware export does not also skip entries within exported data)

[john iv]

Hi Roman, is this going to working finding the \hash directory if you use a softwarelist.xml that's not in the mame.exe location?
e.g. c:\o\games\mame.exe
c:\o\utils\cmpro\software.xml

[Roman]

Nope it currently works only for an exe based listsoftware export and it takes the exe folder as basis for the hashfolder.
So e.g c:/mametest/mame.exe -listsoftware in the xml/exe box then it will use c:/mametest/hash to look for additional files

Was the first idea which came to my mind….open for others

[john iv]

Hmm, does it have to be an .exe extract like 'mame.exe -listsoftware' in the source, or will it take a freestanding listsoftware.xml that has been previously generated, but still in the same directory as the mame.exe?

I like using an externally generated listsoftware.xml as it only appears once in the \fixdats and \scans and doesn't increment.

[Roman]

Currently it does (however it does not always recreate the export but takes the one from the export folder if available) since it needs an indicator where to look for the hashfolder….

As I said it was an idea and a 5 min coding job :-)

[john iv]

Got it, just out of curiosity what are \hash files that aren't accounted for in the -listsoftware output? Are there many?

[Roman]

See screenshot above log list….11

[Shoegazer]

hmm...I've compared coleco.xml from 269 against 270 and there are only 3 changed descriptions. Surely clrmame will show the updated description if the file would be missing...but of course nothing really changes regarding the content of the files....

Wow, looks like I should have checked this more carefully myself.  It seems that in each of those cases, none of the hash values actually changed, only immaterial data like descriptions, and so it makes sense that those sets wouldn't have appeared.  I was especially confounded due to the supracan set being on the list since I was sure there were new supracan software items for 0.270, but it seems there weren't after all.  Sorry for the red herring.

ok...my idea was pretty easy to add:

So in case of an exe based -listsoftware export, the tool now checks if there is a hashfolder for the exe file, takes all *.xml files in there which are not already matching a softwarelist name in the -listsoftware export and additionally loads them to the collection....
works fine...so the only thing which is assumed is that the hash xml filename does match its stored softwarelist name attribute value.....

So...no more overseen skipped software list hashes....(unless -listsoftware export does not also skip entries within exported data)

Thanks, looking forward to testing this once you release the binary. I guess this method is easiest to implement after all?  Will the sets scanned from the hash folder show up in the logs at a specific verbosity?

Also, I ran a more exhaustive search for any sync issues between -listsoftware and the hash directory and couldn't find anything, so I wrote this issue report.  Vas Crabb closed it though, stating no intention to support orphaned lists within the output, so unfortanately it seems another method such as the one you've done in clrmame will indeed be necessary to capture and scan all documented sets. At least we have our answer now with a specific reason given.

[Roman]

The ones from the hash folder are simply added internally, so they are handled as the ones which are already in the listsoftware output. See screenshot above in this thread….the 3do_m2 one at the top is an added one…
Lemme sleep over it before I post a nighly build…maybe I have an idea for John‘s scenario, too

[Shoegazer]

The ones from the hash folder are simply added internally, so they are handled as the ones which are already in the listsoftware output. See screenshot above in this thread….the 3do_m2 one at the top is an added one…
Lemme sleep over it before I post a nighly build…maybe I have an idea for John‘s scenario, too

Yup, and all 11 sets I mentioned are there in the log, your process seems like it works perfectly.
Looking forward to seeing what you come up with.