EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1] 2   Go Down

Author Topic: clrmame and no-intro's header files  (Read 43371 times)

incog

  • Karma: 0
  • Offline Offline
  • Posts: 10
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.0.11 Firefox 3.0.11
    • View Profile
clrmame and no-intro's header files
« on: 13 July 2009, 21:13 »

I've been using clrmame for eons, and it's fantastic! The only annoyance I've ever had with it is trying to rebuild no-intro sets that require their own header files, especially the NES dat.

The problem is that it tends to choke on files that are not even remotely related to the set and the rebuilder will chug away for DAYS using up as much memory and CPU cycles as it desires but never actually getting past the file. So I remove the offending file from my huge un-sorted temp directory and start again, only for it to choke on another unrelated file a few minutes later.

I guess I could manually sort out all the nes files and throw them into their own directory but that defeats the purpose really.

Can you enlighten on why it's choking on these files? The files it most commonly occurs on are large but not stupidly large, but somtimes it will also do the same to smaller files.

Thanks for such a powerful tool. :)
Logged


Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #1 on: 14 July 2009, 05:59 »

if it's not that large you can send me the dat and the romfile and I will have a look....

Generally if you work with enabled header support, each touched file has to be unpacked and analyzed (skip header, calculate checksum) which takes some time...but not really more than a normal deep check like the "unpack and calculate crc32/sha1/md5)" options.

Maybe it's a different option (like "advanced->fix missing scans all files") which touches the file in question over and over again and so you end with a longer scan time.....

so...send me the files in question....maybe plus your cmpro.ini and the used .cmp file which holds settings for the profile (in cmpro's settings folder).
Logged

incog

  • Karma: 0
  • Offline Offline
  • Posts: 10
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.0.11 Firefox 3.0.11
    • View Profile
Re: clrmame and no-intro's header files
« Reply #2 on: 14 July 2009, 15:57 »

Hi, I've isolated a few of the smaller files and uploaded them with the corresponding DAT and Header files, along with my cmpro.ini.

They are up for grabs here.

I tried to rebuild from these isolated files and it worked, they were all skipped as usual which was odd. But when placed back into the larger Temp directory, where they were originally taken from the bug persists. Everything points to a race condition ;_;

To test, add those files to a large directory of files and rebuild using a DAT that requires a header.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #3 on: 14 July 2009, 16:57 »

now I only need to know where to grab the files ;)
Logged

incog

  • Karma: 0
  • Offline Offline
  • Posts: 10
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.0.11 Firefox 3.0.11
    • View Profile
Re: clrmame and no-intro's header files
« Reply #4 on: 14 July 2009, 17:20 »

The link was in the last post, the CSS on this forum just makes the link colour the same as the font colour.

http://www.sendspace.com/file/rlft3n here it is without the colour confusion.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #5 on: 14 July 2009, 17:23 »

ah..silly me...however currently I get a
"Sorry, the free service is at full capacity."
Logged

incog

  • Karma: 0
  • Offline Offline
  • Posts: 10
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.0.11 Firefox 3.0.11
    • View Profile
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #7 on: 14 July 2009, 18:35 »

in the meantime I got the files....now I only need time to have a look at it ;)
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #8 on: 14 July 2009, 19:45 »

ok...first thing....all of the files are NOT fulfilling the NES header constraints.
That's why they are skipped during rebuild. NES files start with a byte pattern of 4E4553 at offset 0. Your files don't.

The slowness if you got them inside a folder with other files is most likely based on the fact that each file in that folder gets decompressed (if zipped/rared/7zipped) into memory, the header detection logic is applied and then the normal rebuilder logic is run on that memory file. But even then it should be rather fast.....

Look at the progress window, it usually tells you which file is currently performed and if it stalls at one file you should have a deeper look at it.
Logged

incog

  • Karma: 0
  • Offline Offline
  • Posts: 10
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.0.11 Firefox 3.0.11
    • View Profile
Re: clrmame and no-intro's header files
« Reply #9 on: 14 July 2009, 20:25 »

I know they are not NES files, they were in a large temp folder I was rebuilding from that contained NES files.

There is no real slowness, clrmame freezes when it hits them inside large driectories but if you isolate those files and attempt to recreate the bug they are simply skipped as they should be, no freeze.

I can leave the process running for days (and have done in the past) and it will never get past a few selective files while they are in that folder.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #10 on: 14 July 2009, 21:02 »

one is rebuilt to "Solar Jetman - Hunt for the Golden Warpship (USA)". I've added 1000 files to the folder and rebuilt...works fine here, no slowdown.
I expect a different file in your temp folder is causing the problem.
« Last Edit: 14 July 2009, 21:06 by Roman »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #11 on: 14 July 2009, 21:10 »

what you can try is to enable rebuilder logging. It's in rebuilder advanced "create logfile". The file may give you more information.
Logged

incog

  • Karma: 0
  • Offline Offline
  • Posts: 10
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.0.11 Firefox 3.0.11
    • View Profile
Re: clrmame and no-intro's header files
« Reply #12 on: 14 July 2009, 21:34 »

Hmm, I think it's due to the massiveness of my temp folder. 8,952 files, 456 folders.

Also thanks to the logging I noticed a pattern I should have right away in those files I uploaded. They are in reverse alphabetical order. The log says they were skipped as they should be.

The first file I removed from that folder because of the freeze was war-1.91.u27, then it was Wagan Land (Japan).gg after another scan, VTECH ..blah... 9344 D.u4 after yet another and finally VS.bin

These were always the last files in the temp folder during rebuilding from it. I never noticed that before. So instead of the stats screen spawning it sits there saying not responding. I thought I just had some corrupt files but it always ends up being the last one alphabetically.

Could it be that rebuilding using headers has some kind of arbitrary limit? I can rebuild them if I throw them into a separate folder and I can rebuild from my temp folder using ordinary header-less dats perfectly.

Thanks for looking into this, I didn't mean to waste so much of your time.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #13 on: 14 July 2009, 21:47 »

hmm..something's odd here. Now it didn't rebuild any file (as before)...I wonder where that "Solar Jetman - Hunt for the Golden Warpship (USA)" came from from a previous try....

will do more checks....

to answer your question: there is no limit. It's the normal rebuilder. The only difference is that the source file is unpacked and run through the header skip routine.....and after that the checksum is compared in the normal rebuilder way of doing it....
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #14 on: 14 July 2009, 21:49 »

hahaha...sillly me...the file came from MAME's playchoice 10 folder which I dropped for some testing ;)

Ok...so here we are again at the start:

No file is rebuilt since none of the files is a NES file (not fulfilling the NES bytepattern rule).

Why do you expect a file Wagan Land (Japan).gg should get rebuilt by a NES dat/header? It's not a NES file.
« Last Edit: 14 July 2009, 21:53 by Roman »
Logged

incog

  • Karma: 0
  • Offline Offline
  • Posts: 10
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.0.11 Firefox 3.0.11
    • View Profile
Re: clrmame and no-intro's header files
« Reply #15 on: 14 July 2009, 22:10 »

I don't expect anything to get rebuilt.

I ran the rebuilder using the NES Datfile through the temp folder and it choked on all 4 of those files in turn. I had to hard kill the process, remove the offending file from the temp folder, dump it into another one and rebuild again. This time it hit the next file, crashed, so I removed that too. x

The rebuilder won't make it to the stats screen if you have a very large folder filled with 8000+ files and a header. It always dies on the very last file.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #16 on: 15 July 2009, 06:23 »

Just ran 10000 sets (non nes) through a nes header rebuild and had no issues at all.

By the way, are you talking about a network drive / samba usage?
Because there are known issues sometimes where wrong configured networks cause timing issues which may result in such a reported behaviour.
If not, you may want to check if virusscanners interfear.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5 Firefox 3.5
    • View Profile
Re: clrmame and no-intro's header files
« Reply #17 on: 15 July 2009, 19:50 »

The issue usually comes with a NAS, network drives and SAMBA.

It has been reported several times and all users came back with something like "My NAS was crap, got a different brand and it's ok now" or "I changed access rights and it's ok now".

So if you use a NAS, network drives or SAMBA, don't be afraid of the message above but give it a try to use a normal hd on your pc.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5.1 Firefox 3.5.1
    • View Profile
idea....
« Reply #18 on: 16 July 2009, 19:23 »

I got an idea, please send me an email, I'd like to send you a special build
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3424
  • Operating System:
  • Windows Vista Windows Vista
  • Browser:
  • Firefox 3.5.1 Firefox 3.5.1
    • View Profile
Re: clrmame and no-intro's header files
« Reply #19 on: 16 July 2009, 21:04 »

ok..what's the idea you wonder...

...well...I always wondered what causes the SAMBA/NAS/USB problems. It's most likely the fact that files are locked. In most cases clrmamepro tried to open files in an exclusive read or write mode. Now if the file is locked, you usually see "file x is used by a different program". The rebuilder skips such files. I've now added shareRead/Write modes....that theoretically should solve the rebuilder weirdness.....
Logged
Pages: [1] 2   Go Up
 

Page created in 0.069 seconds with 21 queries.