EMULAB Forum

clrmamepro [English] => clrmame Discussion => Topic started by: Zandro on 26 January 2013, 07:07

Title: Detector rule: size
Post by: Zandro on 26 January 2013, 07:07

I was attempting to write my first header skipper containing offset rules for files less than a specific filesize, and became frustrated that it was seemingly ignoring them, when it dawned on me that the skipper must be depending on the dat for size info rather than the physical files. The dat does not contain size info, so I must find another method to determine if a file is under a tested size. Is this possible?

There is no particular series of bytes marking the end of 'real data' in these files, and EOF can slide 100s of bytes between files.  I considered forcing an illegal seek to default to EOF, but this also resets start_offset to 0.

I'll be the first to admit that I am less than spry with logic, so I hope you can show me the light with this. Or else confirm there is a limitation that I needn't feel stupid for anymore.  :D

Title: Re: Detector rule: size
Post by: Zandro on 26 January 2013, 14:34

I am also experiencing crashes involving an infinite loop (max cpu) when the rebuilder attempts to scan a smaller file, if no test is given for a header type guaranteeing at least the minimum filesize applicable for the defined range to hash check.

Title: Re: Detector rule: size
Post by: Roman on 26 January 2013, 18:50

send me the header xml plus dat plus example files so I can have a look at that infinite loop...

regarding your original header idea...what exactly do you want to achieve?

Title: Re: Detector rule: size
Post by: Zandro on 27 January 2013, 03:52

PM sent.  I was interested in accounting for these several insufficiently sized files in addition to header skipping normally, and because I found a seemingly insurmountable bug in this situation, think it is better to have those files fixed by the collector to permit equal treatment of all.

Title: Re: Detector rule: size
Post by: Roman on 27 January 2013, 12:55

I will have a look as soon as I find some time....

Title: Re: Detector rule: size
Post by: Roman on 28 January 2013, 20:48

well...ok..the infinite loop was caused by a file where the EOF was reached already but not detected....
http://mamedev.emulab.it/clrmamepro/binaries/cmp20130138_header.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmp20130138_header.rar)

this should be resolved in that build...however I did not have time yet to think about your original task :)

Title: Re: Detector rule: size
Post by: Zandro on 28 January 2013, 21:27

Using rules:
  <rule start_offset="100" end_offset="10200">
    <file size="10200" result="false" operator="less"/>
  </rule>

  <rule start_offset="100" end_offset="EOF">
  </rule>

...Rebuilder now creates undersized test file with no more crashing, but also recreates on each run with statistics repeating "Created Files: 1" and archive timestamp updating.  Scan results statistics is now blank, and scanner wants to remove rebuilt rom for wrong hashes.  I'm just yutzing around at this point, I shouldn't need to use this method in the future.  Re-focusing, moving on..  I appreciate your checking however.

Title: Re: Detector rule: size
Post by: Roman on 28 January 2013, 21:36

I will do further checks (scanner/rebuilder) later...maybe next weekend earliest due to some business stuff to do first.....but at least the EOF loop is fixed :)

Title: Re: Detector rule: size
Post by: Zandro on 28 January 2013, 21:37

This is Odd.
Wrong crc32: rom [wrong: a7ee68fd] [right: ea87ce34]
-- Hash is f80c4a76. ea87ce34 after header skip.
wrong sha1: rom [wrong: 7db857036ac9fba8fa9140d6b13079a86ba75684] [right: ad5e4a4486392aa52220a5bf8dfe64b553c9114b]
-- hash is 21b28e4c0b19908d4948088aea9fcbaef6860d36. ad5e4a4486392aa52220a5bf8dfe64b553c9114b after header skip.

Title: Re: Detector rule: size
Post by: Roman on 28 January 2013, 21:50

yeah yeah...I will look at it next week.... :)

Title: Re: Detector rule: size
Post by: oxyandy on 29 January 2013, 06:34

Hi Roman & Zandro,
A header related thing... so not starting a new thread, when I can hi-jack yours Zandro  ;D

This file is a NES header file 16 bytes long..
http://savage-blades.cjb.net/NES_HEADER_BLANK.hdr (http://savage-blades.cjb.net/NES_HEADER_BLANK.hdr)
Now, if I say try to rebuild a No-Intro NES set (which uses the header skipper),
and this file exists in the Rebuilder source folder,
well, CMP will freeze and never continue past it..

Do you think one day when you have some time, you could look into this ?
Thanks Roman !

Title: Re: Detector rule: size
Post by: Roman on 29 January 2013, 08:12

you used the upper mentioned build already?

Title: Re: Detector rule: size
Post by: oxyandy on 29 January 2013, 10:31

Quote

you used the upper mentioned build already?

No I have not..
Does the build you posted in this thread not have this problem ?
Ok, I will try it later..
Will see..      ??

Title: Re: Detector rule: size
Post by: Roman on 29 January 2013, 10:48

well, http://mamedev.emulab.it/clrmamepro/binaries/cmp20130138_header.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmp20130138_header.rar) fixes an infinite loop.....

Title: Re: Detector rule: size
Post by: oxyandy on 29 January 2013, 11:08

OMG, yes, I just used the version posted in this thread..
You have fixed it !
This is great, the 'infinite loop' getting stuck on 16 byte headers, has long bugged me, have meant to bring it to your attention dozens of times.
 8)
Thank-you Roman !

Title: Re: Detector rule: size
Post by: Roman on 01 February 2013, 21:11

Zandro,

http://mamedev.emulab.it/clrmamepro/binaries/cmp20130201.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmp20130201.rar)

this should fix the remaining "undersized" rebuild/scan issues

Title: Re: Detector rule: size
Post by: Zandro on 02 February 2013, 02:46

Yes, it does :)  However, stats output is still empty.
(https://sites.google.com/site/zandro/statsoutput.png)

With Save LogFile enabled, this is written at the end:

Quote

----------
No statistics are available yet. Please run at least one scan.

I did do New and fast scans...  ::) Maybe some debugging flag was left enabled?

Oxy, do you still get stats output?

Title: Re: Detector rule: size
Post by: Roman on 02 February 2013, 18:34

Stats work fine here

Title: Re: Detector rule: size
Post by: oxyandy on 04 February 2013, 06:28

Quote

Oxy, do you still get stats output?

Ah, I did not see your edit..
Yes I do..
(Roman's Beta builds, must always be added to a fresh download of current 'release' to ensure all files are present)
If you do not have stats.ini present in the folder "Statistics" window will be blank.
see here
http://www.emulab.it/forum/index.php?topic=3622.new#new (http://www.emulab.it/forum/index.php?topic=3622.new#new)

Title: Re: Detector rule: size
Post by: oxyandy on 04 February 2013, 07:21

Zandro,
Can you repeat 'bug' from archive I posted ?
Extract, run CMP, set path to folder apple2gs

New Scan = Merge problem not fixed & Blank "Statistics" output

EDIT: Blank "Statistics" output (caused by missing stats.ini from program folder)

Title: Re: Detector rule: size
Post by: Roman on 04 February 2013, 07:52

I took your package, unzipped it, changed the rompath to the apple2gs folder, changed the 7z path to your provided one and hit scan...it asked me to move files, yes please...worked...and stats are shown, too. So for now I cannot repeat the problem at all...Moving and showing stats work.

little correction...ok...I can repeat the not-move thingie now....however stats are always there....

Title: Re: Detector rule: size
Post by: Roman on 04 February 2013, 08:16

Tested 32 and 64 bit versions under Win7 (64bit ultimate)...always stats..

Regarding the "move" issue, well, it's somehow based in the 7z archive... when I manually do a:

7z d -y -ms=off -mx9 CMP_Header_Fix_2\apple2gs\rastana.7z "rastan (1990)(taito)(us)(disk 1 of 2)[cr2].2mg"
7-Zip [64] 9.30 alpha  Copyright (c) 1999-2012 Igor Pavlov  2012-10-26
Error:
There is some data block after the end of the archive
Nicht implementiert
System error:
Nicht implementiert

In other words: 7z does not remove the file and throws a "not implemented" error....most likely because it's some weird solid archive....

Title: Re: Detector rule: size
Post by: Roman on 04 February 2013, 08:21

...and looking at the archive with a hex-viewer I found the problem the last bytes are "torrent7z_0.9beta".

Again...there is no support for torrent7z...and once again it looks like it does create 7z files which aren't fully supported by standard 7z....

Title: Re: Detector rule: size
Post by: oxyandy on 04 February 2013, 08:50

Hi Roman,
Yes, you got that error with 7-Zip [64] 9.30 alpha

Now, not nit picking, but I supplied 7-Zip [32] 7z 9.28a
Using that, it works perfect..

(http://ndsxdelta.no-ip.org/apples_with_apples.png)

I checked resulting archive manually, archive is updated no problems there.

Title: Re: Detector rule: size
Post by: Zandro on 04 February 2013, 09:26

Right, blank stats, even with my test involving raw data going into zip archives (I can't see what that has to do with it anyway). Roman, are there any diagnostics/debugging methods I can run for you?  Running Win7 HP SP1 x64.

Title: Re: Detector rule: size
Post by: oxyandy on 04 February 2013, 10:35

Sorry Roman,
I bet Zandro has done exactly the same thing as I have...
hahaha
Zandro, in your CMP folder, do you have a copy of stats.ini
in there ?
No ?
Get a copy from CMP 4.09a and put it there  ;D

EDIT however, I do have more to report on 7z merging issue, will post on appropriate thread..

Title: Re: Detector rule: size
Post by: Zandro on 04 February 2013, 10:37

Quote

Zandro, in your CMP folder, do you have a copy of stats.ini
in there ?
No ?
Get a copy from CMP 4.09a and put it there  ;D

WHAT IS THIS BLACK MAGIC

Title: Re: Detector rule: size
Post by: Roman on 04 February 2013, 14:25

Actually I had the torrent7z error also with 7z 9.20...but anyway...in the meanwhile I found the reason why files do not get removed (and so the move failed).....and I think the stats issue was based on your missing ini file...

Title: Re: Detector rule: size
Post by: oxyandy on 04 February 2013, 16:25

Quote

and I think the stats issue was based on your missing ini file

Absolutely, lesson learnt = the hard way...

Always put Roman's beta builds, (exe)
into a folder with the files supplied with the release build..

Thanks for fixing things again, and so quickly !