Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Shoegazer]

In scanning software chds over a network share, during the "Looking for unknown sets" phase, cmpro is taking a very long time, some of the larger sets can take 12 hours or more to complete.  It was never like this before, even with this version, so I'm wondering if I inadventerntly turned something on that casued this but I couldnt find anything. The scanning process involves adding all software xml dat files from the MAME hash directory and then scanning all of them in batch mode.

Any suggestions are welcome.

* OS: Linux Mint 22 / kernel 6.8 (via wine 9)
* CMPro version: 4048d
* MAME version: 0.268

[Roman]

Sounds weird. If it takes such long you either have a major network problem or chd files are read as standard files so that some hash calculation over the whole file is used, but actually for chd files this shouldn't be the case, it would simply read the sha1 values stored in the chd headers which should be quick.
 
Try to minimize the problem by running a scan on one profile which takes long, try to reduce the amout of files for that profile (e.g. reduce the available files to just one chd) etc...
And the typical "did you turn it off and on again" thing is on the test list, too :-) Or a fresh profile for one of the long duration ones.

[Shoegazer]

Thanks, Roman. I did try resetting my network device and rebooted which seemed to help somewhat but it's still very slow.  For example, now testing the "saturn" set and it takes about 10 seconds per chd, and those are from 200-500mb.  I guess during the "Looking for unknown sets" phase it has to scan each chd just to verify that the sha1 is listed in the datfile?  And is there a way to turn that off, or at least speed it up through a more basic check?

By the way, the non-chd sets scan quite rapidly.

[Roman]

"looking for unknown sets" is just a simple dir walk and hash compare and for chds it only reads 128 byte of the .chd file. So there is no "read the full file" when it comes to chds.

I've just tested it over the network with one chd which has a wrong name and one with a right name and a completely different one which does not appear at all in the datfile.
The "looking for unknown sets" was less than 1 second for such 3 files.....

You can send me your cmpro.ini plus the *.cmp file of the used profile if you like so I can take over your settings

[Shoegazer]

Thanks so much, Roman. I've sent you a PM with the files.

[Roman]

Thanks for the files but testing leads to the same result.
It's fast. I can't and won't test it under Wine though,plain Windows 11 and a network folder as rompath (\\fileserver\bla...).

As I said: When reading a chd file only 128 byte are read to get the header and the stored sha1 values from it and if you say it worked way better before, maybe something else (e.g. Wine) has changed. Sorry, but I don't have an idea and can't repeat the problem.

Give the new scanner a try.

[Shoegazer]

Fair enough, well thanks for trying. Nothing has changed on the samba/smb end so I'm not sure where the differences lie, but I'll just deal with it, maybe I'll try running the scans at night or something but it's unfortunate when you're used to something much faster (I was getting the speeds you observed).  The new scanner doesn't work in Wine, but hopefully if the source can be opened there may be some interest in a native Linux port.

[Roman]

hmm...it doesn't work in WINE? Why so? Any error message?

[Shoegazer]

Actually nevermind, it was a configuration issue on my end that I fixed and it works under Wine now. 

At first glance this seems to be a much cleaner and more intuitive interface than cmpro, honestly it looks great.  I assume the scanner and rebuilder will be merged at some point.  This will be fun to use and I'm hoping to eventually replace cmpro that I've been using since the late 90s.  I realize your new project is still in early stages so I'm not sure I want to fully replace it just yet though.  I'll test with a few sets and software lists first.

By the way, in your readme you indicate an objective of no Windows-specific code - what are you using for windowing (fingers crossed for Qt?).  And do you have any interest in porting to native linux yourself or would that be left to others once the source is opened?

[Roman]

The UI is currently more or less a throw-away ui...which was quickly done in (yuck) MFC.
The separated core is stl /c++ 20, plus 3rd party libs which are portable.

Yes, one of the next steps is to combine scanner and rebuilder and add some profiler. Currently I'm doing some smaller things first, one minor thing which was found during MAME's .268 cylce and 32k path length support. After that I try to combine the 2 modules and after that the profiler will come....

[Shoegazer]

Okay tell me how you really feel about MFC But I'm with you.  Perhaps something like Qt for the long haul if you find time for it.  Nice to hear the core is portable though.  Also, I'm sure you're aware of zlib but other cross-platform projects use it partially because it's patent-unencumbered as they state, just something to consider vs. ZipArchive.

Looking forward to the improvements down the pike regarding the profiler etc.!  How cool.

[Roman]

Well, I mainly use ZipArchive because I'm used to it and it supports copying data (single/multi files) from one archive to another without recompression ;-)

[Shoegazer]

Ahh yes I suppose that would be critically important to speed.   I looked here to see if zlib can accomplish this but nothing popped out at me, though that doesn't mean it isn't there?  I realize it's a rare use case though.  Perhaps if it really isn't, the zlib team might entertain adding it.

[Roman]

Yep, it's all about speed ;-)
Actually it's not hard to code at all (we're not talking about solid archives here), it's just some modifying header entries and moving data blocks....Before using the zipArchive class routines I had an own routine back in the 90s ;-)
I still hope that bit7z (currently used for 7z and rar) might include it at least for zip so I'd be able to only have one 3rd party lib for archives....

[Shoegazer]

Hah, why am I not surprised you've been that far ahead of the curve?

I wasn't even aware of blit7z, interesting. It's even Mozilla PL 2.0 which I assume would be fine by you.  Heck I'd be happy to add a feature request there for what you need if you think it would help, though from what you're saying I suppose that kind of work is already underway.