Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[mimoluo]

This happens several times, and I suspect there is a bug in the scanner which misinterprets the same rom name of different systems as unneeded.

I have a complete arcade rom collection 0.238.
Now trying to incorporate SL chds into the collection with one single profile.

The system PATHs are individually defined.

The "new scan" function in "scanner" asks permission to remove unneeded files in the "fix unneeded file" window.
After the scanning, the report shows that my complete arcade collection turns into an incomplete one, in which thousands of ROMs are reported as missed.

Some of them can be found in the backup folder.
Some of them gone forever.

This is not the first time I bumped into this problem.
At first, I thought that was my misconfiguration.
Now I can replicate the problem by adding SL (MESS) CHDs into the corresponding software folder and do a new scan.

enclosing my cmp file FYI.

[Roman]

Well, can you give a single example about the "unneeded" messages? Or one of the thousands of missing roms messages?
I highly doubt that something gets lost since all unneeded files are put into the backupfolder...so please give an example here, too.

Your setup (split of chds and sets in different folders) won't work in SL combined mode unless you really only have Cubo BIOS sets and TaitoGNet files in the chds folder (since that's what you've setup in your config). For the combined mode, you need one unique path for the given bios/standard/device...so e.g. for Area51, you need area51.zip in the 'standard' assigned path and also a subfolder in there named area51 which contains the chd.
Keep in mind, chds need subfolders named after the set, that also happens for software lists.


So....I need more information about your setup....some examples which files are placed where and why message is shown......first impression is that the split of chds/roms into different folders is simply not what you can use in the combined SL mode....
If you want to keep it, you should switch to 1-profiler-per-sl mode.

[mimoluo]

1. I unified the BIOS/Standard/Device paths into .../ROMs
2. encosing to scan logs (before and after) which may be differentiated to tell what part was lost. (mostly software listed roms)
3. attaching an image showing a rebuilder recreating thousands of the lost files.
4.  including the prompt to remove an unneeded file in the "scan" after the rebuilder

[mimoluo]

Further enclosing a log of the scanning when I refuse to "remove the unneeded roms" but keeping them.

The report now indicates that I miss 0 roms.
----------
Missing
·Sets                 7538/174315
·ROMs                 0/543182
·CHDs                 8688/11652
·Samples              0/19344
·Bytes                0/435gb

[Roman]

Only had time for a quick view on it yet...

The fix unneeded screenshot looks good to me since it mentions a cpc_flop softwarelist set ("ace of aces (uk) (1985) [original] (weak sectors).dsk" ) which does not belong to a a7800 assigned rompath (I can imagine that the a7800 corresponds to the Atari 7800 software list).

In your "after" list I can see lots of "move to rompath" messages which all belong to pippin SL sets...Again, the message is correct since it seems that you've placed the pipping chds with their subfolders in a rompath which was assigned for standard MAME ones, not Software lists. Since the rebuilder does not move/create chds, they must have been moved there on a different way...maybe manually....

The other files now listed as missing in "after", well, you said you only did a rebuild in between....maybe such files were part of the dropped/source/addpath? Please double check the settings. If a new scan removed them, you'd be prompted, so please read carefully what the prompt says.


So, please double check your folders (a7800 folders shouldn't contain cpc sets) and try to minimize the problem to 1 set where you can say: It was correctly stored before the scan and it was removed completely afterwards. By the way...do you scan a network drive or anything else which may be special to your setup?

For example, focus on "11 A Side Soccer (UK) [softwarelist: cpc_cass - folder: 11assoc - size: 55kb]" (1st entry in "after" after the pippin messages) which was not listed in "before" but as missing in "after". Try to repeat that behaviour that "it disappears". Either the rebuild did something to it (but that can only mean that you used it as rebuilder source + remove rebuilt...but then it should exists in the destination) or the scanner, but then you should see a prompt.

[Roman]

ok, let's take that the set "aceoface" with the one and only rom "aceofaeu.bin"....

where was it placed before? (O:\Games\Emulator\Mame\Software\a7800\ ?)
Was it compressed/archived (.zip/.7z/.rar) or decompressed?

You can limit the scan to that only set (set information dialog, uncheck all, only enable that one), so it would be interesting to know what the scan message shows....if it was removed there should be some fix unneeded prompt...I need the detailed information on that.

You're saying "amout of missing roms seems random"...which is actually a bit scary since it more sounds like a totally different issue (3rd party programs interfearing like virusscanners locking/blocking files or network connections failing (see SAMBA posts here on the forum)...so...can you give some detailed information about your setup. Is  O: a network share?

[mimoluo]

I rebuilt the complete rom set and redo the scan with only a7800\aceoface
It is placed at O:\Games\Emulator\Mame\Software\A7800\ in a zip file.
O:\ is totally local drive.

The prompt comes again, I accepted 3 removals and denied 6 of them.
But the report shows 0 missing roms this time.
This leads to two questions:
1. Why does the rebuilder rebuilds the roms but the scanner wants to remove them.
2. The removed roms are not reported as missed this time, this leads to further works to identify those being removed and reported missed. (which may be a long list to test)

Is that because I choosed the "merge" mode which may be confused between different systems ?

BTW: the removal this time creates a backup file in the backup folder, named aceoface.zip comprising the 3 removed roms.  But even weirdly, running a rebuilder from the backup path does not recreate anything from the backuped aceoface.zip.  So they are truely  unneeded?

[mimoluo]

Not telling lie, I scanned all sets after the 1-set scan, and accepted all the removal.
The report turns into 0 missing.
Running rebuilder from the removed files in the backup folder does not create anything new, confirming that they are truely unneeded.
Still no technical explanation yet.
----------
Missing
·Sets                 7535/174315
·ROMs                 0/543182
·CHDs                 8685/11652
·Samples              0/19344
·Bytes                0/435gb

Fixed Wrong Case
·Sets                 0/0
·ROMs                 0/0
·CHDs                 0/0
·Samples              0/0

Fixed Unneeded
·Sets                 0/0
·ROMs                 481/481
·CHDs                 0/0
·Samples              0/0

[Roman]

(This is actually a reply to the post before.....you've updated the forum in between....)


Several things happen here:

the scanner is right in removing the files since (according to the names) they aren't a7800 files at all, they are cpc_flop ones and so the scanner is fully right with removing them since they don't belong there.

backup is correctly created, so no files are lost.

"the removed ones are reported as missing"...well, maybe because they are missing for the cpc_flop software list

"running a rebuilder from the backup path does not recreate anything". Can be various reasons: e.g. the files do exists in their right destination already...rebuilder log could be useful.


But there is one main question:
where do the cpc_flop sets in the a7800 folder come from?
Did the rebuilder put them there? Are you sure that before rebuilding they are not in the a7800 folder and the rebuilder puts them there? Since I can't repeat that behaviour at all with the settings at all (7800 files are rebuilt to the assigned 7800 folder and cpc_flop to the assigned cpc_flop one) it's hard to say what is going on on your system.

Maybe you used a different profile for rebuilding with different settings?
Double check the assigned folders for cpc_flop and a7800.

Can you please send me (again): your *.cmp file for the used profile, cmpro.ini

The merge mode should not matter at all.

[mimoluo]

After the last scan that I agreed to remove 500ish roms, the report shows 0 missing.
Then I rebuilt the total sets from mirror again, no new files created.  (this took me 8 hours)

However, odd things happened.
In the "audit all sets" in mameui.exe, roms including 1943 are confirmed affected.
To this step, I can confirm that one of the previous scans actually affects the arcade roms, but the report shows false result.

Using "new scan" in the scanner to double check, new odd behavior occured.
new unneeded roms are asked to be removed, including 1943 in the arcade ROM path.
(enclosing the removed 1943.zip FYI)

Looks like a confusion between arcade and software list.
and I wonder why "new scan" and "scan" get different results.

path problem should be excluded.  the config file looks fine.
and I stick to one profile at all time. 
Now I may decide to separate the SL and arcade into two profiles to get rid of this mystery.

In the "new scan" this time, it asks to add missing rom.
This is new.

[mimoluo]

The "new scan" is ongoing, and asks to rename wrong names.
This is newly occured situation, too.

[Roman]

Sorry but posting screenshots doesn't really help. Please focus on one set as I said before.
Don't get me wrong, I really appreciate your testing, but we need to find something which can be repeated in a small scenario.
If you get random effects, something totally different (even maybe outside of clrmamepro) might cause it. So try to limit scans, maybe do a clean cmpro installation or setup a new clean profile from scratch.

From what you've said and showed, I see no problems with the scanner since it always reported things which do make sense.

The old scenario was:
you've got a rather good sets with maybe some files missing, you do a rebuild and another scan then shows unneeded files which of course should not happen. And here it is important to only look at one of now unneeded files. Did it exists before the rebuild (hopefully no) and did it appear after the rebuild (but before the scan). This information would be important since it would clearly show that the rebuilder added the files.

In the concrete example it was an ace of aces set (a cpc_flop SL set) which was rebuilt to a path which from the name implies to be the a7800 SL list.
Surely the scanner afterwards complains about it that it's unneeded there.

So the question is: Is this small scenario repeatable? And if so, why does the rebuilder put files there. I've tested that and I can't repeat it. I will do another round with your latest provided cmp/ini files (thanks for that).

[mimoluo]

Thanks.  Will try to narrow down the issue.

BTW, the "new scan" is concluded, rendering a total different report, wherein I accepted every fix it proposed.  It is indeed random and scary.

Missing
·Sets                 7953/174315
·ROMs                 9235/543182
·CHDs                 8687/11652
·Samples              664/19344
·Bytes                2gb/435gb

Fixed Wrong Case
·Sets                 0/0
·ROMs                 0/0
·CHDs                 0/0
·Samples              0/0

Fixed Unneeded
·Sets                 0/0
·ROMs                 96/96
·CHDs                 10/10
·Samples              0/0

Fixed Wrong Name
·Sets                 0/0
·ROMs                 5/5
·CHDs                 0/0

[Roman]

Random indeed sounds weird.

I can tell you what I did for testing:

- clean cmpro installation (4.044 is current )
- added a profile "mame 0.238" based on official mame.exe, imported the data with all SL enabled, all prompts answered with OK/YES
- quit cmpro, generated all your "O:" paths, replaced cmpro.ini and mame 0.238.cmp with your files
- put your provided aceoface zip (which actually includes all a7800 and cpc_flop versions) in your rebuilder source folder
- ran cmpro again, did a scan...everything was missing...well...sounds good
- ran a rebuilt
- ran a scan again, nothing is unneeded and the rebuilt files were ok (cpc in the cpc folder, a7800 ones in the a7800 path)

Did that several times....it is always ok...

Maybe you can setup such a clean install, separated scenario, too....maybe on a different drive...that "random" thing sounds really a bit like something is weird...maybe just bad cache data...

[mimoluo]

Thanks.
Took 12 hours falling back to step 1.

1. isolate SL from arcade.
An arcade only profile is created to rebuild and scan the rom sets.
7000+ unneeded Roms are surpringly moved out of the Rom path ...\ROMs
It appears that the previous processes actually merge the arcade sets and SL sets into one parent.
That might explain why there are so many random confusions because SL and arcade overlap so much.
The independent arcade rom sets are now cleaned and confirmed 100% correct in mameui.exe

2. Will do SL in independent folders and profiles to see if things still happen. (to be continued)

[Roman]

"It appears that the previous processes actually merge the arcade sets and SL sets into one parent."

And that's the only problem and open question you need to find an answer for.

You're always saying you did a scan which ends ok, then a rebuild and then another scan which shows unneeded (such "wrong merged" sets). So it seems that the rebuild operation added the files, but that's why I asked if the files don't exists before and are clearly created by the rebuilder (before the new scan and anything in between). We need to be clear on such steps before checking anything else.

The only thing I can think of is that you used the "MERGER" instead of the rebuilder in between...which can be the cause of this since that module is outdated and not really aware of SL.....

[mimoluo]

updating the lastest findings.

using mameui.exe to generate two profiles.
1. one without sl,
2. one with sl.

scanning the same rom sets.
the first one is 100% ok.
In the second profile, only enable to scan the basic "standard/device/machanical/bios/" but no "SOFT", and the scanner removes pwrshovl, and reports missing rom/chd of power shovel.

I hope this unique case is duplicable to find the source of glitches.

[Roman]

You can't have profiles based on the same mame.exe where one is with and one is without SL!
You will run into cache issues since the cache key is based on the binary's hash value, so the one without SL will load the cache information from the one with SL or vice versa (depending on which was created first). The only possibility would be to always clean the cache before loading one of such profiles. Every time!

And again, please stop creating new scenarios and focus on:
https://www.emulab.it/forum/index.php?topic=6337.msg21264#msg21264

[mimoluo]

I found that the collection is totally spoiled by the said two-profiles when I woke up, before I see this post.  Sincerely feeling sorry about that.
Will be back when there's any useful update.

[mimoluo]

The only thing I can think of is that you used the "MERGER" instead of the rebuilder in between...which can be the cause of this since that module is outdated and not really aware of SL.....

Yes, I found that after merger, things go wrong again, even if I only maintain SL roms.