Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[B2K24]

When scanning with AntoPISA's AllProjects DAT CMP starts to crawl when scanning the snap and titles and shows "not responding" indefinitely. If you use Process Monitor you can see the CMP process is producing I/O related errors, while scanning the files multiple times.

Roman, if you wouldn't mind having a look, I've put together a reproduce-able package where you just have to extract the zip (Datfile inside) then just set rompath to CMP TEST and execute a new scan.

https://www.sendspace.com/file/fsf3tl

[nimrodeo]

@Roman,

Compare the results when the "Missing" checkbox is checked on the Scanner dialog.

With it unchecked, no problem.  With it checked, the problem is apparent.

[Roman]

Nothing's wrong here.

It simply takes pretty long since it finds thousand of "missing but fixable files" after processing each set (especially all the place holder screenshots for devices, etc etc). If you open the scanner tree window you see how many entries are added and keep added throughout the scan.
You can turn off this "missing but fixable files" check in scanner->advanced->deeper check for fixable missing files. But even with it turned on, the scan finished after roughly 5 minutes. Turning it off of course ends way way faster.

Generally, scanning artwork sets (no matter if archived or unpacked) takes rather long due to the way how they are organized (1 set with multi-ten-thousand-roms)

[B2K24]

Understood Roman, thanks for taking a look and answering

[nimrodeo]

@Roman,

Thanks for looking into this.  If I understand correctly, it is the massive number of "roms" in each "machine" of AntoPISA's AllProjects .dat file that clrmamepro must slog through in order to finish checking for any of the missing items.

What if the AllProjects .dat were to be subdivded into a smaller set of .dat files and sent through clrmamepro's batch mode?  Would that be likely to have an impact on performance regarding checking for missings?

I know I should probably test it myself, but I just thought I would check with you to see if you thought it could speed things up.

[B2K24]

AntoPISA already does individual dat files for most if not all of that content.

[nimrodeo]

AntoPISA already does individual dat files for most if not all of that content.

Exactly.

But has anyone ever timed the difference of the single all-in-one vs. the batchmode of the individual .dat files?

It seems to me there might be the difference of 200,000 x 200,000 tests vs 20 x 10,000 x 10,000 tests, since 40,000,000,000 > 2,000,000,000.  It might go up to 40 times faster.

[Roman]

There are some settings/options which are time eaters. SHA1 calculation, the lookup for missing roms which I've mentioned before etc..etc..
Solid 7z files, unpacked files...or hardware related things like slow hds, too litte diskcache etc...

When having some of them in mind, you can lower the processing time immensly (e.g. turn off fix-missing and the lookup for fix missing and use the rebuilder to add files).

I don't think there is a big time difference in scanning the All-Dat Progetto dat versus the single ones (unless they'd share a lot of files). Of course you got more data in memory if you use the all-dat one and lookup operations may took longer simply due to the amount of data...but nothing that time consuming.