EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: CMP hangs with process produces I/O related errors  (Read 4804 times)

B2K24

  • Member
  • *
  • Karma: 2
  • Offline Offline
  • Posts: 29
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 49.0 Firefox 49.0
    • View Profile
CMP hangs with process produces I/O related errors
« on: 03 October 2016, 04:15 »

When scanning with AntoPISA's AllProjects DAT CMP starts to crawl when scanning the snap and titles and shows "not responding" indefinitely. If you use Process Monitor you can see the CMP process is producing I/O related errors, while scanning the files multiple times.

Roman, if you wouldn't mind having a look, I've put together a reproduce-able package where you just have to extract the zip (Datfile inside) then just set rompath to CMP TEST and execute a new scan.

https://www.sendspace.com/file/fsf3tl
Logged


nimrodeo

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 11
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 53.0.2785.116 Chrome 53.0.2785.116
    • View Profile
Re: CMP hangs with process produces I/O related errors
« Reply #1 on: 03 October 2016, 08:33 »

@Roman,

Compare the results when the "Missing" checkbox is checked on the Scanner dialog.

With it unchecked, no problem.  With it checked, the problem is apparent.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 53.0.2785.143 Chrome 53.0.2785.143
    • View Profile
Re: CMP hangs with process produces I/O related errors
« Reply #2 on: 03 October 2016, 20:13 »

Nothing's wrong here.

It simply takes pretty long since it finds thousand of "missing but fixable files" after processing each set (especially all the place holder screenshots for devices, etc etc). If you open the scanner tree window you see how many entries are added and keep added throughout the scan.
You can turn off this "missing but fixable files" check in scanner->advanced->deeper check for fixable missing files. But even with it turned on, the scan finished after roughly 5 minutes. Turning it off of course ends way way faster.

Generally, scanning artwork sets (no matter if archived or unpacked) takes rather long due to the way how they are organized (1 set with multi-ten-thousand-roms)
Logged

B2K24

  • Member
  • *
  • Karma: 2
  • Offline Offline
  • Posts: 29
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 49.0 Firefox 49.0
    • View Profile
Re: CMP hangs with process produces I/O related errors
« Reply #3 on: 03 October 2016, 21:15 »

Understood Roman, thanks for taking a look and answering :)
Logged

nimrodeo

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 11
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 53.0.2785.116 Chrome 53.0.2785.116
    • View Profile
Re: CMP hangs with process produces I/O related errors
« Reply #4 on: 03 October 2016, 21:43 »

@Roman,

Thanks for looking into this.  If I understand correctly, it is the massive number of "roms" in each "machine" of AntoPISA's AllProjects .dat file that clrmamepro must slog through in order to finish checking for any of the missing items.

What if the AllProjects .dat were to be subdivded into a smaller set of .dat files and sent through clrmamepro's batch mode?  Would that be likely to have an impact on performance regarding checking for missings?

I know I should probably test it myself, but I just thought I would check with you to see if you thought it could speed things up.
Logged

B2K24

  • Member
  • *
  • Karma: 2
  • Offline Offline
  • Posts: 29
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Firefox 49.0 Firefox 49.0
    • View Profile
Re: CMP hangs with process produces I/O related errors
« Reply #5 on: 03 October 2016, 21:52 »

AntoPISA already does individual dat files for most if not all of that content.
Logged

nimrodeo

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 11
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 53.0.2785.116 Chrome 53.0.2785.116
    • View Profile
Re: CMP hangs with process produces I/O related errors
« Reply #6 on: 03 October 2016, 22:01 »

AntoPISA already does individual dat files for most if not all of that content.
Exactly.

But has anyone ever timed the difference of the single all-in-one vs. the batchmode of the individual .dat files?

It seems to me there might be the difference of 200,000 x 200,000 tests vs 20 x 10,000 x 10,000 tests, since 40,000,000,000 > 2,000,000,000.  It might go up to 40 times faster.
« Last Edit: 03 October 2016, 22:05 by nimrodeo »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 112
  • Offline Offline
  • Posts: 3287
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 53.0.2785.116 Chrome 53.0.2785.116
    • View Profile
Re: CMP hangs with process produces I/O related errors
« Reply #7 on: 04 October 2016, 07:25 »

There are some settings/options which are time eaters. SHA1 calculation, the lookup for missing roms which I've mentioned before etc..etc..
Solid 7z files, unpacked files...or hardware related things like slow hds, too litte diskcache etc...

When having some of them in mind, you can lower the processing time immensly (e.g. turn off fix-missing and the lookup for fix missing and use the rebuilder to add files).

I don't think there is a big time difference in scanning the All-Dat Progetto dat versus the single ones (unless they'd share a lot of files). Of course you got more data in memory if you use the all-dat one and lookup operations may took longer simply due to the amount of data...but nothing that time consuming.
Logged
Pages: [1]   Go Up
 

Page created in 0.14 seconds with 20 queries.

anything