Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[McHale]

I've been using clrmame pro since almost the beginning and am very comfortable (and fond) of it.  I hadn't kept my roms and chd's up to date for over a year and decided this weekend to finally merge everything in.  After more than 12 hours of scanning it's only at 1%.  It completes only 3 or 4 per hour and the roms it's taking for ever on aren't ones currently in my set so it's not a matter of unzipping anything. 

Here is my current environment:

I just downloaded the latest clrmame pro a couple days ago and am running the latest version.  I'm running it against the official 0.252.  I'm attempting to run a full scan checking roms, sets, samples, and chds.  I have everything under "fix" unchecked, everything in the advanced fields are set to the defaults, "show stats" and "ask before fixing" are both unchecked.  All of my files are of the .7z variety (all have been maintained by clrmame pro).  Clrmame pro is installed locally on my C drive while all of the roms are on an external drive.  There are no apps running on my PC except for clrmame pro.

I have cancelled and restarted the process a couple times (before I counted the past 12 hours for this current scan), even rebooting the PC in between restarts.  I don't know what else I can check or change to improve the speed. 

Any help would be greatly appreciated. 

[Roman]

A normal scan on a (more or less) modern drive usually takes (depens a bit on the sha1 settings) just a couple of minutes, so yes, something is wrong...

I'd go with a clean install (so all default options are set) and then simply do a scan without any fix options enabled. That should also be pretty fast and you already get an impression what is wrong with your collection. If too much appears, you may even have setup wrong rompaths...or a wrong merge mode (and fixing tries to recreate all the sets...). So checking the first non-fix scan would be a good idea to see if something is totally wrong configuration wise.

After that you should add missing stuff with the rebuilder and not via fix-missing (so keep that disabled).

Solid 7z archives may become a problem as soon as write operations are done on them since it would require a complete recompression of the files.
3rd party tools like virusscanner can interfear. If you're using a network drive, bad configurations can cause slow downs...even bad drivers can do.
A soon dying harddisk can also be a slow down factor....but don't worry...pretty rare case...


....and sure I recommend to use an official MAME import, no softwarelists, split sets and zipfile usage

[McHale]

I built the xml file from the official .exe so that's OK.  It was definitely a clean install  (unzipped to a download folder and manually run from the folder - not in a system folder so it isn't a rights issue).  When I initially ran it I had all fix options checked and it did repair a handful of roms (as verified by the file date and time).  It was while I was doing this that I saw how it merged files in.  Unzip, add one file, rezip.  Unzip, and the 2nd file, rezip, etc. etc.  Brand new 4TB drive. 

The fact that it's hanging up during the analysis of the file and not while interacting with an existing ROM file is what has me stumped.  I paid attention to the files it was going slow on and ended up in the "Scan Results" window and if they existed in my ROMs folder I moved them out to see if it would fly by.  It didn't.  As of right now, I'm still at 1%.

My MAME cab that I've always done this on is down so I'm merging using my 2 year old laptop with the hard drive in an enclosure.  Copying large amounts of data back and forth go pretty fast so I assumed it wasn't the drive.  I'm going to disable the Microsoft AV (which I didn't even think of as it's probably trying to scan each and every file) but I didn't think that would interfere with the original scan for files that don't exist. 

A real head scratcher.  As always, thanks for the help.  If and when I figure out the problem I'll report back as an FYI.  I assumed I was the only one having the issue and it wasn't a bug but it's good to get confirmation.

-Mc

EDIT:  Disabling Antivirus didn't make any difference.  Now to do some drive tests...

EDIT 2: I forgot to mention I'm on Windows 11 (where my MAME PC was Windows 7).  I'm waiting for clrmame pro to cancel so I can see if the firewall settings could somehow be an issue as well.

[Roman]

Operations are file not set based, so e.g. the rebuilder will - when it finds a hash match - transfer the one single file to the destination. As long as you're using zip files, this is done without recompressing the files (unless you turned on recompressing). For rar/7z files it's different, there, data will be temporarily unpacked and added. Solid 7z files are a pain, since it will require a recompression of the whole archive over and over again. So yes, zip (or decompressed sets) are fast, 7z/rar is slow, solid 7z archives are very very slow.

As I said, a normal scan with fix operations disabled should be fast since no archive operations are involved. If this already takes hours for you, you have a different problem. To give you a number....a first full MAME scan on a 6 year old i7 machine is approx. 5 minutes...a second one where most of the disk access is cached, 1 minute, using a HD (not SSD).

The normal (non-fix) scan tree output should already give you an impression of the state of your collection. If nearly each and any set is listed with issues, you may have chosen a wrong merge mode or you're scanning a wrong path which would result in thousands of archive changes....so double check before running fixes.

General speed tips:

- using the rebuilder to add new files is better than using fix-missing, since fix-missing looks up several sources and is slower in the end
- fix missing can be a time killer (e.g. in progetto snaps sets)

- use zip files (when using 7z or even solid 7z, a post operation may be better which then recompresses touched files in one go)
- use split sets
- turn off sha1 checking
- know from where to where you do rebuild operations, same disk can crawl (in this context, the cmpro temp folder is also a thing to take into consideration)

- for rebuilding, use the new standalone rebuilder ;-)

[McHale]

When you first start the scan you know how it does the "testing unneeded" first?  It lists 2 roms per second.  When it's doing the scan, it's clearly finding the existing roms as I'm watching the current set/description fields and noticing the games like 1941, 1942, 1943, etc are passing with no issues so that confirms the folder path is correct. 

When I look at the Windows Task Manager it shows almost no usage for CPU, Disk, or Network.  Typically 0% CPU and 0.1 MBS Disk speed.  It's like it's simply running in slow motion.

Windows Compatibility mode did nothing either.  I'm going to install a 2nd copy of clrmame pro and run it against empty folders on the local HD and see how fast that goes.  If it runs normally the issue looks like it is related to the external drive.

[Roman]

It lists found files, not roms, archives most of the time. This is done by a standard tree walk, so nothing special. It does not print out any set but maybe each 10th or 100th, can't remember.
Again a number for:...that "looking for unknown sets" at the start takes about 12-15 seconds (depends a bit how many rompaths you have).
During a scan I see HD activity at about 50% active time and between 5 and 20MB/s read speed (most of the time 5-9 with some small hits to 20).
That's on Windows 11, I7-8700K, 16GB and a Seagate IronWolf 4 TB HD

0.1MB/s sounds way way too slow...

No idea what's going on on your system. Try a different drive, a different machine, just to see how it performs there...(if possible)

[McHale]

Using the exact same clrmame pro install I created another profile with dummy folders and no contents and it flew through the process.  So it's not the machine or the install. 

It has to be the drive.  It's a new drive and the health is fine but who knows.  I found another 4tb external drive I'm copying everything over to do all of the merging then I'll moved it back to the other (slow) drive once it's done and see what the MAME performance is like.

Thanks again.  I still don't know specifically what the issue is but I'm pretty sure I know generally. 

-Mc

[McHale]

Yep.  I moved everything to a different external drive and it's almost done doing the entire merge.  It literally took longer to copy everything across than it will to do the merge. 

Thanks again for the help and the great app. 

-Mc

[Roman]

Thanks for the feedback....sounds weird for a new drive...hmm..maybe it has an encryption process enabled (bitlocker etc)?