EMULAB Forum
clrmamepro [English] => clrmame Discussion => Topic started by: joelkolb on 16 October 2020, 17:01
-
I am working with an older set, 0.160. I already had a complete set of 0.159 and used n 0.159->0.160 upgrade set with the rebuilder to get to 0.160. I did a scan afterwards to make sure everything is good. The statistics show that I'm not missing any sets, CHDs or samples and nothing needed to be fixed, but it shows that I'm missing 3 ROMs. The scan results window is completely empty so I have no idea what those ROMs are. It's driving me nuts. Is there any way to figure out which ROMs are missing?
Here's a screenshot:
(https://i.imgur.com/RIndETd.png)
-
First of all, don't care about stats. The only important thing is that the scan results tree is empty.
Use the latest nighly or official build, clean the cache in the profiler, check the following options and do a new scan.
You should first check if you don't hide anything and check all sets. All "you want to scan" options are enabled and all "check" options are enabled. Scanner->SetInformation->click "select all", in Scanner->Systems, select "all", in Scanner->Hash & CHD->Available Regions are all checked, in Scanner->Scan Results Tree->Context Menu->View->Hide fully missing sets should be disabled.
All these are default options, so unless you've changed them they should be ok.
If all of these natch your settings and you still get 3 missing in the stats it maybe something weird in the old Mame versions. There is no real reason to use anything but the current mame release.
-
You should first check if you don't hide anything and check all sets. All "you want to scan" options are enabled and all "check" options are enabled. Scanner->SetInformation->click "select all", in Scanner->Systems, select "all", in Scanner->Hash & CHD->Available Regions are all checked, in Scanner->Scan Results Tree->Context Menu->View->Hide fully missing sets should be disabled.
All my settings are just as you described except for "Scanner->SetInformation->click "select all"". I don't have a section called Set Information in the scanner. What am I missing?
-
I found the Set Information at the bottom of the Scan Results window. All sets were already selected.
-
and now update to MAME .225 and don't use a 5.5 year old MAME version ;) or simply turn off stats.
-
At least I know I did everything right. Thanks for your help.
-
I can prepare a version for you which logs the 3 ones into a textfile to check which weird things happened on your place...
-
try https://mamedev.emulab.it/clrmamepro/binaries/cmproTest.7z
It creates a cmpro.log file in the cmpro main folder and lists all missing roms in there. Would be interesting to see which are listed there. The file is locked until you quit cmpro and emptied on startup.
-
Did you give that version a try with your old MAME version? Would be interesting to see which roms are marked as missing
-
I did. Here are the contents of cmpro.log after the scan:
hng64\bios_export.bin
hng64\bios_korea.bin
hng64\bios_us.bin
With that information I was able to track down those 3 ROMs and get a clean scan that showed 0 missing down the line in the stats.
I'm a bit of a perfectionist so having this info is great, IMHO. I wish this were part of the main release versions. Thanks again for all your help!
-
But they definetly should appear as missing in the scan results tree if they were really missing (and all hide this, hide that, disable this/that was turned off).
I will do some tests
-
Did some tests with 159 and 160...and I can't reproduce your problem. If the files are missing, they are listed in the tree.
-
I was also cleaning up an 0.174 set last night and the same kind of thing happened with that. The results pane was empty but the stats showed 3 missing sets and 58 missing ROMs. Unfortunately, I didn't save the cmpro.log from that but it was consistent with what the stats showed and after gathering the ROMs listed a subsequent scan showed 0 missing in the stats.
Just now I finished updating a set that was a few versions old to get it up to 0.225. This was done on a different computer from the other 2 with a new setup of clrmamepro. I'm getting the same deal. Missing 1 set and 37 ROMs in the stats but the scan results are empty. Screenshot and cmpro.log with duplicate entries filtered out are below.
(https://i.imgur.com/bcSG9ap.png)
cmpro.log contents:
aristmk5\setchip v1.0.u11
aristmk5\setchip v1.0.u7
aristmk5\setchip v4.03.07.u11
aristmk5\setchip v4.03.07.u7
aristmk5\setchip v4.04.05.u11
aristmk5\setchip v4.04.05.u7
aristmk6\01040505.u70
aristmk6\01040505.u71
aristmk6\01040505.u83
aristmk6\01040505.u84
aristmk6\03130334_left.u70
aristmk6\03130334_right.u83
aristmk6\14011605_left.u70
aristmk6\14011605_right.u83
aristmk6\20012305_left.u70
aristmk6\20012305_right.u83
aristmk6\25012805_left.u70
aristmk6\25012805_right.u83
atarisy1\136032.105.l13
atarisy1\136032.107.b2
atarisy1\136032.115.j10
awbios\fpr-24363.ic48
bubsys\boot.bin
cdibios\zx405042p__cdi_slave_2.0__b43t__zzmk9213.mc68hc705c8a_withtestrom.7206
crysbios\mx27l1000-alt.u14
konendev\2v02s502_ifu.u190
megaplay\315-5349a.ic54
megaplay\315-5651.ic8
megaplay\315-5653.ic56
megaplay\315-5655.ic34
megaplay\315-5661.ic7
naomi2\epr-23607c.ic27
naomi2\epr-23609b.ic27
neogeo\uni-bios_4_0.rom
segasp\fpr-24329.ic72
stvbios\epr-17741a.ic8
tourvis\v4-54.ic29
-
Do you really miss the files in the log file or do they actually exist in your paths?
It's pretty normal that you need to run two scans to cleanup some stats/listed entries.
There are some rare cases where a first scan puts parts to the backup folder and a second one will add them back to their correct places. ...but still weird that you get an empty results tree and entries in the stats....never seen that one though.
can you send me your:
cmpro.ini
and the used dat (in case you didn't import from a mame binary) and the beloning *.cmp settings file for your used profile (in cmpro's settings folder).
-
Yes, all of those specific ROM files are missing from the sets except for konendev which is the 1 set indicated in the stats that is missing, so of course the ROM for that is missing also.
I did have to run several scans. Just as you said, several ROMs that were needed were moved to the backup folder and showed as missing in the results. A rebuild with the backup folder as the source and another scan fixed those. I also ended up with problems with several CHD files in a subsequent scan. It told me that they were being detected as v4 and v5 was needed. It was very odd because when I checked them with chdman they were all v5. I don't remember which files specifically but they were mostly variants of Street Fighter III. After another scan none of those issues with the CHDs came up. Altogether I had to do about 4 or 5 scans and manually finding a few missing files in between each scan before it came back as clean as it did here. In every case I've mentioned where the scan results were empty but the stats showed missing ROMs, I did multiple scans and got things to the point where I could do several scans or new scans in a row and still get them same exact results.
I have attached the files you requested with .txt appended as the extension. There's no dat because I used the MAME 0.225 binary to make the profile. You will notice a lot of references to the folder "MAME_0.205_Non-Merged" because I was upgrading from 0.205 when I started.
-
The chdversion problem sounds even more weird.
Since the chd file holds the version which is simply read and compared to the one which you've set in settings->compressor->chdman->req chd version.
Does it always show the same results or random?
If random it more sounds like a system problem where something is interfearing, where "something" can be anything from bad network drivers when the data is stored on a network drive, bad RAM/HD, virusscanner or other 3rd party tools interfearing etc..etc..
Don't worry, sounds more dangerours at it is....but such errors were never reported before.
-
The CHD version thing is random. I've never had that happen before and when I did another scan right after it didn't say anything about the CHDs anymore.
-
Can you tell me a bit about your system?
Which OS (version, 32/64bit), I can see that you store your data on drive D...what kind of drive is that?
If things happen randomly there is a chance that something interfears. There were cases in the past where bad samba network drivers caused issues or virusscanners simply blocked access and lead to weird results. Not to mention overclocked systems where RAM simply produced flipped bits here and then.
I don't say that your system has a problem but if you got the chance to test stuff on a different drive or machine, it would be interesting to see the results there.
But of course I will double check the chd version check (surely a not correctly initialized variable may cause random issues, too). But again.....no one else reported such kind of things..... ;-)
-
One thing I've spotted in your settings is Scanner->Advanced -> you got "Separate BIOS sets" disabled. This means you waste pretty much space by keeping the bios files in each and any set instead of keeping them separated (e.g. each neogeo set keeps the neogeo bios files instead having 1 neogeo.zip holding them). Disabling that option is very very very rarely used...actually so rarely used that I can imagine it's the cause of your stats issues. I will do some checks on that option but you should turn that option on. The same option exists in rebuilder->advance.
And you're using non-merged sets...again, not a good idea. You waste a lot(!) diskspace and actually nobody uses this ;)
The common modes are split merged or fully merged sets (split sets / merged sets).
So...really think about switching your merge mode to something common ;-)
Regarding chd versions...no luck here to repeat the behaviour yet.
-
ok...I can now for sure tell you that your stats/scan results tree mismatch is based on the disabled "separate bios sets" option.
If this mode is enabled and the bios sets itself misses files (but all the sets using this bios got the bios files included already) show no entry in the scan results tree but stats count them as missing.
This is of course a wrong behaviour. It should list the bad bios set in the tree. I will check why it does not appear (for whatever reason cmpro might saw the existing files in the single sets and didn't pop out the bios itself).
Thanks for helping me by sending your config.....but you should really think about split merged mode and separated bios sets....you will gain diskspace.
-
OS is 64 bit. The one that was working with 0.160 and 0.174 is Windows 10 (2004) and the files are on a mapped network share from an Unraid storage server. The one working with 0.225 is Windows Server 2019 and the files are on a locally attached Storage Spaces volume. Built in Windows Defender for A/V for both machines. No overclocking in either case. Both very different systems and storage but same type of results with clrmamepro so I don't think it is a system problem.
The settings you mention are intentional. I know it takes up a lot more space but fortunately I have lots of space. I really like having each game self contained without dependencies on any other files (mostly, except for CHDs and samples). So I do set clrmamepro for non-merged and do not separate BIOS files. Also, in some front ends like Emulation Station you can either set it to only show files in a precompiled XML list or to show all files in a particular folder. I like the flexibility of being able to quickly drop a ROM in the folder without having to edit the XML and have it show in the front end but I hate having BIOS files showing alongside the actual playable games for MAME. By not separating the BIOS files so that I don't need to have a separate BIOS file in the folder along with the games I can have things work just how I like them.
I'm glad that you were able to pinpoint the cause of this behavior with missing BIOS files and the stats and scan results. Hopefully you will be able to figure out how to fix it. I am happy to help. Thank you for working on this with me.
-
Well, I actually thought that your intention was to use not merged sets ;-)
...but there's a lack in your plan. Device sets. Nearly each set requires device sets, too...and they aren't put in a non merged sets and currently they are not handled like non-separated-biossets.
I already got requests to add something similar for device sets but had no time yet to add it.
For sure I will double check the missing tree print out very soon.
-
here we go: https://mamedev.emulab.it/clrmamepro/binaries/cmpro20201026a.7z
-
I tested this version and I'm a bit confused as to what it's doing. I used it to do a scan of my 0.160 set expecting that this time the stats would show 3 ROMs missing like before and the scan results would show me the 3 missing ROMs instead of nothing. But it actually fixed the set that was missing the ROMs. When I did a new scan everything came back clean with 0 missing and when I looked in the file hng64.zip it had all the files including the 3 that were missing before. Just to double check I made a backup of hng64.zip and deleted the 3 ROMs from the zip. I then did another full scan and it came back clean again and when I check the file the 3 ROMs were there. I'm not sure where it is getting these 3 ROMs from to fix the set. Also, shouldn't there be an indication in the stats that the set was fixed?
-
So what's confusing? fixing missing files works exactly like that.
hng64 missed 3 files and cmpro found the files in other places, so "fix missing" automatically added them and so hng64 won't appear in the scan results tree since it's ok now. If you don't turn on the fix options and you got the 3 files missing in hng64, you will see hng64 in the tree appearing. So it works as expected since the tree result only shows problems. The stats most likely shows something like 3/3 missed/fixed whatever.
cmpro looks in addpaths, backup folder and parent/clone relationship sets for missing files and in your case it found them somewhere there....
-
OK. I guess I was confused because it didn't fix the set in the previous version even though it was set to fix automatically. I was looking at it more like the problem was in the information it showed during and after the scan (stats and scan results) and not that it wasn't fixing files that it should have been fixing. That makes sense. If it fixes everything then there won't be any issues at the end. The stats did not show anything fixed though. It must have gotten the 3 missing files from clones because when I did the last scan with the copy of hng64.zip that I broke on purpose I also made sure the backup folder and add-paths were empty. Thanks!
-
I am getting the same results with my scans. I am using split sets (which I need to convert) but I get 20 missing ROMs but no listing of them. I understand you have looked at this problem already and it probably has to do with BIOS sets. I am using 0.226 MAME and cmp4039a_64 so everything is up to date there. One other problem is that the listing of "Missing Sets" now takes two days to complete when it used to take a minute.
Thank you for your update to cmp4039a_64 because I was having monumental problems with clrmamepro for a few months. But, cmp4039a_64 seems to have fixed all of my problems except for the BIOS sets. Thanks for all you do.
-
try https://mamedev.emulab.it/clrmamepro/binaries/cmpro64Test.7z
It creates a cmpro.log file in the cmpro main folder and lists all missing roms in there after a scan (and leaving cmpro). Would be interesting to see which are listed there. The file is locked until you quit cmpro and emptied on startup.
The code for the "missing sets" hasn't changed for ages and it's simply a directory walk/crawler over the files so it should be pretty fast. Depending on your amount of files and disk speed it should indeed only take up to a minute...but never 2 days. Sounds like some other app (maybe virusscanner) is interfearing....or something is locking/blocking access.
-
It turns out that the files were not bad but in another directory parallel to my directory called mame_bios_soft:
\
|-mame_bios_soft in ROM-Paths
|-mame_bios_soft_bad not in ROM-Paths
|-mameroms in ROM-Paths
|-mameroms_bad not in ROM-Paths
Files were moved before the scan run from a correct directory (mame_bios_soft) to the parallel directory (mame_bios_soft_bad) that contained bad ROMs from previous scans. I keep bad old ROMs around for awhile just to make sure they are not needed. Of course those files had correct Checksums but names that had an extension added like crimep2_11.zip instead of crimep2.zip. This was done by clrmamepro in previous scans.
So now you know what the problem is when there are zero ROMs listed in the error list but ROMs showing as Missing. Certainly another weird thing that is still in clrmamepro.
-
Can you show me the cmpro.log output and not the missing list please.
There is a reason why they are not listed in the scan results tree (but counted as missing). Actually I more expect the stats to be wrong than the output.
The miss list you've posted has most likely nothing to do with your missing stats. The "miss list" works totally different than the scan. As mentioned it is just a quick set lookup by file/folder name. It does nothing in detail at all. A relict from 20 years ago. Actually nobody should use that ;)
The only important information is the scan results tree output. Even statistic count values are not really important. But of course it is interesting to see which files are counted as missing here....
-
cmpro.log is empty, even after ending cmpro. Yet it still shows files missing in the summary and nothing in the scan. Should I be using the version you provided in this thread?
-
Yes, you need to use https://mamedev.emulab.it/clrmamepro/binaries/cmpro64Test.7z
Simply replace the 64bit binary (keep a copy of the official one).
-
Yes I did finally use the test version and it had the same folders with the numerical extensions that came from the bad list (that took 2 days to produce), expanded to the individual files:
crimep2_11/cp2_1.dat
crimep2_11/cp2_2.dat
crimepat_12/cp_1.20_u1.bin
crimepat_12/cp_1.20_u2.bin
crimepat_14/cp01.dat
crimepat_14/cp02.dat
fastdraw_130/fast_01.bin
fastdraw_130/fast_02.bin
gallgall_21/gg_21_rom1.bin
gallgall_21/gg_21_rom2.bin
lastbh_006/bounty_01.bin
lastbh_006/bounty_02.bin
maddog2_100/md2_1.0_1.bin
maddog2_100/md2_1.0_2.bin
maddog2_110/md2_1.10_u1.bin
maddog2_110/md2_1.10_u2.bin
maddog2_202/md2_01.bin
maddog2_202/md2_02.bin
maddog_202/md_2.02_u1.bin
maddog_202/md_2.02_u2.bin
maddoga/maddog_01.dat
maddoga/maddog_02.dat
spacepir_14/sp_14_u1.bin
spacepir_14/sp_14_u2.bin
wsjr_15/wsjr151.bin
wsjr_15/wsjr152.bin
zortonbr_100/zb_u2.bin
zortonbr_100/zb_u3.bin
I have searched every directory that could have those files and found nothing. The versions of the files that I had around before, I moved to a completely different disk that is not shared with the computer that is running clrmamepro. So, I am at a loss as to where clrmamepro is finding those files. These files are all from alg_bios if you don't recognize them.
-
Did you check if you enabled all sets and all systems?
Maybe you got an option enabled that hides things. So please check the following and do a new full scan.
Scanner->Systems-> ensure that all checkboxes are checked (or hit all)
Scanner->Scan Results->Set Information->Ensure that all sets are selected (e.g. by hitting "select all")
Scanner->Scan Results->Context Menu ->View -> ensure that "hide fully missing sets" is NOT enabled
Scanner->Hash & CHD->Available Regions -> all checkboxes should be checked
If you like you can also send me your cmpro.ini file plus the belonging *.cmp file for your profile in cmpro's settings folder (I assume you're using an official MAME binary as data source)
-
Yes all my sets are selected. All systems are selected including alg_bios. All set information is selected. Hide fully missing sets is NOT enabled. All regions are selected.
Files attached
-
Thanks a lot for the files. I've found the problem for not listing the files. Very nice finding.
So update to 4.040
-
Glad you found it. I am a professional tester for HW/SW/FW etc. so I usually find things like this.
Additional:
When I ran 4.040 I had some problems. I did Rebuilder to add the missing files, then did a Scan. Mostly it was fine, but one thing that happened was that the first scan said the files were in where they should be in alg_bios but it showed that they were in mameroms also. I searched for the files in mameroms and could not see the files. I ran the scanner again and the second time it was correct with no files missing and no extra files in mameroms. I am very happy now. Thanks for 4.040 and all you do.
-
If you got dupes (identical sets exist in various rompaths) after running a rebuild, you may have rebuilder settings which don't match the scanner ones (e.g. you don't use sysdefpaths in rebuilder while you do in scanner). So a set might get rebuilt to a not wanted place accidently.
Anyway....Most likely the scan run cleaned them up. There are some rare cases where a second scan run can fix some stuff which a first one popped up.
Nothing really to worry...since your 2nd run seems to resolved it.