Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Shoegazer]

Recently I've observed that with the 4.x versions of CLR, deep-scanning CHDs (i.e. with "Decompress CHD and check MD5/SHA-1" option flagged) takes longer than I recall with the 3.x versions - perhaps even twice as long.  Is this a known issue with recent builds or could this be on my end?  I don't have any other "Hash and CHD" tab options flagged but maybe there's something else to tune for better performance?  Thanks.

[Roman]

i will check that next week.

[Shoegazer]

Thanks, Roman.  Would be great to know what you find.

[Roman]

So...I did some quick test with area51.chd.

Testsystem: 64bit Windows7, 4GB, C2D6600
Deep CHD scanning and all check options enabled but only area51 selected

chdman -verify : ~0:22 min
clrmamepro 3.x: ~1:01 min (I was aware of that lag and always wondered why it's so slow)
clrmamepro 4.01: ~0:26 min (nearly the same as pure chdman plus some little overhead for unneeded set check, UI refresh etc).

So...I don't see any lag in the cmpro 4.x versions anymore which is good since I always wondered where the poor performance in 3.x came from.

The only difference might be the used zlib code. I didn't test 32bit versions yet but for now I don't see anything wrong in 4.x ...actually I see an improvement over 3.x

[Shoegazer]

Wow, not sure what's wrong here then.  Are you aware of any performance differences between deep scanning to a remote WinXP/NTFS drive vs. Linux/ext4 over SAMBA?  It's the only other thing I can think of, since I think I was using the former when I noticed faster results. 

[Roman]

NTFS no, SAMBA yes.

Actually SAMBA was always a problem and I have no idea why. SAMBA causes the rebuilder to not scan all files from the source and you need multiple rebuild operations to get all files scanned for example....sounds like timing problems.... Don't ask me why...some user said he got rid of this problem by setting some access rights. Others reported no issues at all...

so...SAMBA is evil

[Shoegazer]

Thanks for the info, Roman - even if it wasn't the greatest news at least I understand a little more what's actually happening.  By the way, it sounds like you are referring to general incompatibilities with CMP and SAMBA, not just CHD deep scanning, right?  I didn't notice any issues with SAMBA other than this performance issue, and scanning was pretty quick without the "deep" option flagged, though now that I think about it, I haven't rebuilt anything with that setup either.

Anyway, it seems there are only a few options to attempt resolving the performance problem:

1) Attach the ext4 drive to a Win7 system and use something like this to read it.

2) Use CMP with Wine on a Linux system with the ext4 drive attached (any issues that you know of there?)

3) Wait for CMP to become a little more Linux fs friendly (and not I'm not about to bug you about that, but wait I just did, ooh see what I did there?) 

Thoughts? 

[Roman]

well, yes, the descibed samba issues are more rebuilder related.

how long does chdman -verify take on your system for 1 chd (e.g. area51.chd) compared to cmpro (e.g. select area51 in setinfo to only scan that set)?

cmpro uses the same code as chdman....so I really wonder why there is a big difference (like there was in cmpro 3.x).

[Shoegazer]

Ok, benchmarks:

CHD deep scan area51.chd (CMP 4.01): 10mins 55secs
chdman -verify area51.chd: 2mins 54secs

So yes, considerable difference there, assuming those two operations are equivalent in function (both decompressing and checking CHD against hashes).  Again, based on what you said earlier I'm now thinking CMP 3.x builds could exhibit the same behavior, though I haven't tested with any of those lately (though I could if you think it would help). 

Either way this is odd CMP behavior considering it uses the same code as chdman as you mentioned.

[Roman]

Well as I mentioned, I had no idea why there was the delay in cmpro3.x. The only change for that in 4.x is actually a newer ziparchive library and it's unicode compiled. I'm actually happy that the result is now back to the chdman performance.

There is however one difference between cmpro and chdman. cmpro runs the hash calculation (of the decompressed blocks) in own threads...Maybe this causes a massive overhead on your system of something goes wrong with the sync (however it works fine here...so it's coded correctly ). I may be able to create you a version with no threading....but don't ask me when...pretty busy week in front of me and then some little holiday. But I try to create it some evening this week.....(no promise thuogh)

[Shoegazer]

Sure, Roman - I'd like to test out a single-threaded build when it's convenient for you to make one.  I'm curious to know if that's the issue. 

In case it helps, I'm running CMP 4.01 on Win7 x64 with a Sandy Bridge mobile proc (i7-2620M).  I also have an older C2D system I could try it on (as I understand it, threading models are completely different with the C2D architecture).

By the way, did CMP always have multi-threaded hash calculation?  If not, I could also try downloading/testing an old build that doesn't.  Also, are there any other features in CMP that are multi-threaded?

Thanks again, Roman.

[Roman]

On your sandy bridge you get > 2min to run" chdman -verify area51.chd"? Wow..that's already slow, even if it's a network drive....(ok...but still cmpro should be roughly the same as chdman)

yes, there are several other parts (e.g. all archive reading ) which are threaded....

[Shoegazer]

Yeah, I was a little surprised myself about that.  It does seem a tad snappier than the C2D, though not as much as was hoping.

[Roman]

I will see what I can do....

[Roman]

ok, here's a 4.01 version which does not use threading for chd deep scan...let me know if it has any effect (here it's maybe 2 seconds speed gain for area51)

http://mamedev.emulab.it/clrmamepro/binaries/nothread.rar

In the meantime I did some tests with 3.x but nothing really was discovered as bottleneck. Every single part seems to be slow (or at least not as fast as expected)...which is weird since in 4.x it's as fast as chdman (at least here ). So...maybe gcc does some way better optimization than VS2008.

[Shoegazer]

Thanks, Roman.  So I plunked your single-threaded cmpro64.exe in the install directory and ran it.. but the results are about the same: area51.chd took over 10 mins to scan.  For shiggles I tried the 32-bit CMP you included but got basically the same results as expected.  Likewise, I tried the same scan with CMP 3.138a - similar results.

I ran chdman again and it just blazes through the verify (relatively speaking of course).  Not sure what else would help explain the difference between cmpro and chdman though.  Realizing they both share the same code, are you sure chdman -verify actually decompresses and checks against both hash types as CMP does?  Maybe it's only checking one of them (after all, it reports "SHA1 verification successful" when it's done).  And maybe there's something about the way it decompresses?  Also, I used the chdman from MESS .143 to compare, in case it matters.

[Roman]

cmpro uses the chdman source code, so functionally, it's identical and actually the verify process is pretty straight forward. Loop over all hunks, read them in, decompress them (if needed), calc sha1/md5/crc32s...

The only changes are:
1) the zlib decompression and crc32 calculation points to the ziparchive class' zlib module (it's plain zlib as in MAME)
2) the OS dependent I/O routines point to windows api calls
3) the sha1/md5 calculation is threaded and uses C++ classes for MD5/sha1
4) cmpro is compiled with VS2008 and not gcc

I did already some tests with several changes to the above setup with the 3.x build...however, no real bottleneck was found. Generally it's slow. It might be a combination of slow reading (maybe chdman does some better buffered reading) and compiler settings...so maybe 2) and 4) are the main reasons. What I still wonder is that 4.01 acts so fast compared to 3.x (on Windows system, no network drive) while the only difference there is the updated ziparchive lib and an unicode build.

[Shoegazer]

Thanks for the explanation.  Just to keep you up to speed: I ran a test against the C2D system to check for differences but got similar results. :/

I hate to return to this, but what about the hash algorithms?  If the CMP option is to "Decompress CHD and check SHA1/MD5", whereas chdman simply says "SHA1 verified successfully" after it's done, then at least at the surface it looks like CMP is doing a bit more; i.e. also performing a calc of MD5.   This would at least explain things, however my logic is probably flawed.

Also, perhaps there's something going on during the download.  Both tools have to download the chd from the network share, but maybe chdman streams it differently?  I tend to doubt the 802.11n network is the bottleneck with CMP but maybe it is?

Also, I suppose if you have time to create a build with GCC, that would at least eliminate item #4 from your list.  Just a thought.

[Roman]

look at the verify routine in chdman (chd.c) and you will see that chdman calculates both hashes md5/sha1. Actually cmpro already does better here since it only calculates hashes which are needed (newer chd versions only store sha1).
Easy test for you: copy the chd to a local drive and test it there. With chdman and clrmame.

However, I do have a slowdown in cmpro 3.x with local drives but it's not related to hashes or networkdrives. But 4.x is as fast as chdman so I can't really say it's gcc/hashes/drive based (it could be zlib though...since that was updated in 4.x).

gcc compile is no option, cmpro is a vs2008 project and I don't want to spend weeks to get it running
with gcc (if even possible).

I may find some time to look a little bit more deeper into that 3.x/linux slowdown of November. On the other hand, 4.x is fast on the prefered platform