EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1] 2 3 4   Go Down

Author Topic: Logging of Rebuilt Source Files  (Read 31346 times)

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Logging of Rebuilt Source Files
« on: 17 July 2014, 13:34 »

Hi Roman,

I am suddenly in dire need of a logging feature when Rebuilding.
1. The name of the Source archive.
2. The internal file-name. (Which matched the dat)
3. What they were Rebuilt to.

I have never played with CMP logging previously, so does something like this exist already ?
Many thanks
Oxyandy
« Last Edit: 17 July 2014, 13:50 by oxyandy »
Logged


Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 80
  • Offline Offline
  • Posts: 2702
  • OS:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 36.0.1985.125 Chrome 36.0.1985.125
    • View Profile
Re: Logging of Rebuilt Source Files
« Reply #1 on: 17 July 2014, 13:43 »

Yep...exists already...

Rebuilder->Advanced->Create logfile

....

creates something like:

C:\Users\Roman\Desktop\circus.zip\circus.2a
  -> springbd.zip\93448.2a
  -> circus.zip\9005.2a

Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #2 on: 17 July 2014, 13:53 »

Ok,
Many thanks for the quick response..
Where do I find log ? Guess cmp program folder root.
With each run, is log appended or starts new ?
I guess I'll find out soon enough..
I'll try now
Thanks again.

EDIT: Ok all answered, I give it a run
« Last Edit: 17 July 2014, 13:54 by oxyandy »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 80
  • Offline Offline
  • Posts: 2702
  • OS:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 35.0.1916.153 Chrome 35.0.1916.153
    • View Profile
Re: Logging of Rebuilt Source Files
« Reply #3 on: 17 July 2014, 17:26 »

as you might know by now...you can specify a full path to the file...and right beside that input field you got a checkbox for appending/overwriting the file.....
Logged

oddi

  • Member
  • *
  • Karma: 2
  • Offline Offline
  • Posts: 178
  • OS:
  • Windows NT 6.3 Windows NT 6.3
  • Browser:
  • Firefox 31.0 Firefox 31.0
    • View Profile
Re: Logging of Rebuilt Source Files
« Reply #4 on: 17 July 2014, 17:39 »

O my god! Me continuous learn CMP :)
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #5 on: 17 July 2014, 17:43 »

Yes, I saw that in Rebuilder Options as soon as I setup "Logging"
I see is a per 'dat' log & not global, but that's ok..
I just finished writing a program to reformat CMP's log output to the 'exact' format I require.
Has all the info I need, just perfect

Thanks again
Oxy
Logged

oddi

  • Member
  • *
  • Karma: 2
  • Offline Offline
  • Posts: 178
  • OS:
  • Windows NT 6.3 Windows NT 6.3
  • Browser:
  • Firefox 31.0 Firefox 31.0
    • View Profile
Re: Logging of Rebuilt Source Files
« Reply #6 on: 17 July 2014, 17:45 »

Oxy what is that your program?, give me any private beta for test :)))
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 80
  • Offline Offline
  • Posts: 2702
  • OS:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 36.0.1985.125 Chrome 36.0.1985.125
    • View Profile
Re: Logging of Rebuilt Source Files
« Reply #7 on: 17 July 2014, 18:12 »

if you got any requirements for the layout of the file...let me know.....
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #8 on: 17 July 2014, 19:07 »

Hi Roman,
Really I am very happy already, no need for you to change anything.
Already making use of logging to delete "Source Files" while keeping a reference list of the matching "File-names"
Output from my 'CMP Log Parser' is like this:
Code: [Select]
New: "File-name matching dat.ext"  Old: "File-name found when Rebuilding.ext"
This lines is kept if (New: & Old:) name differs greatly, otherwise is filtered from the "New_Log.log" I make.
It also gets alpha sorted & merged to a Master.log

Thanks for the kind offer.  ;D

Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #9 on: 18 July 2014, 04:18 »

D:\My Documents\Downloads\小霸王SB-A8 NES解密版\小霸王 SB-A8-NES\WXNDecrypt\三侠五义.wxn [SKIPPED Reason: no hash match]

D:\My Documents\Downloads\小霸王SB-A8 NES解密版\小霸王 SB-A8-NES\WXNDecrypt\三侠五义.wxn.nes

I am not sure why lines like these have no [REASON ?
They no longer exist (oops probably should have done a trial run first)
They create a quirk in my log, which I can re-write code to handle..
Sadly right now I can not repeat it, but I assume is related to previous file-name ?

EDIT: (previous file-name ?) No, I guess not.
EXAMPLE: A single archive with 521 internal files = 8 files skipped with-out [Reason.
Seems almost random.. I need to get more familiar with what's happening exactly how logging works.  :-\

Anyway Roman, no problem, I just have more learning to do..
I need to learn what causes output lines like these.
My program starts with a 10Mb CMP_Log.txt after parsing by my exe, it is split into 3 separate logs

1. 178Kb ROM log
in this format
Code: [Select]
New: "100m Hurdles (Waixing).nes" Old: "110米栏.wxn.nes"
New: "1943 - The Battle of Midway (U) [!].nes" Old: "1943 (U).nes"
New: "2011 Tank by Lei Fan (Battle City Hack).nes" Old: "2011坦克_leiyin.nes"
2. 178Kb Source Archive/Folder log.
(needed cause something internal names are poor, compared to archive name)
3. 9.7Mb Remaining files, which I need to write a filter for (extensions I want to be alerted about)
« Last Edit: 18 July 2014, 15:53 by oxyandy »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 80
  • Offline Offline
  • Posts: 2702
  • OS:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 36.0.1985.125 Chrome 36.0.1985.125
    • View Profile
Re: Logging of Rebuilt Source Files
« Reply #10 on: 18 July 2014, 07:13 »

I will have a look why (or when) no reason is shown....
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #11 on: 18 July 2014, 07:31 »

I am trying hard to make a simple repeatable scenario, but having trouble with that:
For example I scan the whole D: drive.
After I look in the log.
D:\Folder A\Archive_A.zip is causing 'missing reasons'

So if I drag & drop Archive_A.zip onto Rebuilder, everything is OK.
If I set Source D:\Folder A\ in Rebuilder, again everything is OK.

trying hard here....
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #12 on: 18 July 2014, 07:36 »

Hang on, an obvious check, I will disable my AV and try over whole drive again.
That made no difference so it seems, it still looks like it's happening... sigh
« Last Edit: 18 July 2014, 15:46 by oxyandy »
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #13 on: 18 July 2014, 08:02 »

D:\Desktop\nesroms\不能运行的[vt].zip\高速赛车.wxn.rar [SKIPPED Reason: no hash match]
D:\cmp4014__32\temp\高速赛车.wxn_11449.rar\高速赛车.wxn.nes [EXISTS: Highway Racing I (Ch) (Wxn)\Highway Racing I (Ch) (Wxn).pas]


D:\Desktop\NES_TOOL_TEST_ROMS\New Folder (5)\不能运行的[vt].zip\高速赛车.wxn.rar [SKIPPED Reason: no hash match]
D:\cmp4014__32\temp\高速赛车.wxn_9813.rar\高速赛车.wxn.nes

Hard to repeat, seems random.. will keep searching for "Repeatable Event"

EDIT:不能运行的[vt].zip seems to be quite repeatable so I copied it to several locations.

« Last Edit: 18 July 2014, 15:44 by oxyandy »
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #14 on: 18 July 2014, 08:12 »

Roman,
The ONLY clue I seem to have is that the [Reason - only seems to be missing from files which actually match the dat.
These should be [EXISTS:

D:\My Documents\Downloads\南晶科技.rar\南晶科技\[NJ057]魔界塔士.zip [SKIPPED Reason: no hash match]
D:\cmp4014__32\temp\南晶科技\[NJ057]魔界塔士_29345.zip\card_cover.jpg [SKIPPED Reason: no hash match]
D:\cmp4014__32\temp\南晶科技\[NJ057]魔界塔士_29345.zip\Mo Jie Ta Shi (C).NES
D:\cmp4014__32\temp\南晶科技\[NJ057]魔界塔士_29345.zip\box_cover.jpg [SKIPPED Reason: no hash match]
« Last Edit: 18 July 2014, 15:43 by oxyandy »
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #15 on: 18 July 2014, 08:22 »

I got it !
I seem to have a "Repeatable Scenario" now.
I will upload a folder & a dat.
See if you can repeat also..
I will PM link to files - ATM Dat is private. :)
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 80
  • Offline Offline
  • Posts: 2702
  • OS:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 36.0.1985.125 Chrome 36.0.1985.125
    • View Profile
Re: Logging of Rebuilt Source Files
« Reply #16 on: 18 July 2014, 14:19 »

well..yes...it happens when a file was already rebuild....so if A is rebuild to a1,a2,a3...and then A comes again it skippes the reason...(it should show EXISTS then....)...fixed...
http://mamedev.emulab.it/clrmamepro/binaries/cmp20140718.rar
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #17 on: 18 July 2014, 14:30 »

Thanks very much Roman !
Now I can make some very quick use of this.
I have LOTS of files to deal with ;)
I was just going to point out, the exact thing you said..
After a file has been 'match once already' - it happens
Code: [Select]
D:\Desktop\New Folder (19)\0_2D Escape (Ch) (Wxn).zip\2D Escape (Ch) (Wxn).nes [EXISTS: 2D Escape (Ch) (Wxn)\2D Escape (Ch) (Wxn).nes]

D:\Desktop\New Folder (19)\1_2D Escape (Ch) (Wxn).zip\2D Escape (Ch) (Wxn).nes

D:\Desktop\New Folder (19)\2_2D Escape (Ch) (Wxn).zip\2D Escape (Ch) (Wxn).nes

« Last Edit: 18 July 2014, 14:30 by oxyandy »
Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #18 on: 18 July 2014, 14:37 »

Confirmed Roman,
Working perfectly :D
Code: [Select]
D:\Desktop\New Folder (19)\0_2D Escape (Ch) (Wxn).zip\2D Escape (Ch) (Wxn).nes [EXISTS: 2D Escape (Ch) (Wxn)\2D Escape (Ch) (Wxn).nes]

D:\Desktop\New Folder (19)\1_2D Escape (Ch) (Wxn).zip\2D Escape (Ch) (Wxn).nes [EXISTS: 2D Escape (Ch) (Wxn)\2D Escape (Ch) (Wxn).nes]

D:\Desktop\New Folder (19)\2_2D Escape (Ch) (Wxn).zip\2D Escape (Ch) (Wxn).nes [EXISTS: 2D Escape (Ch) (Wxn)\2D Escape (Ch) (Wxn).nes]

Logged

oxyandy

  • Member
  • *
  • Karma: 4
  • Offline Offline
  • Posts: 265
  • OS:
  • Windows XP Windows XP
  • Browser:
  • Firefox 29.0 Firefox 29.0
    • View Profile
    • .
Re: Logging of Rebuilt Source Files
« Reply #19 on: 20 July 2014, 18:24 »

Hi again Roman,
Logging is working great,
I have a problem with "Header Skipper XML's"
Code: [Select]
<?xml version="1.0"?>
<detector>
  <name>No-Intro NES Dat iNES Header Skipper</name>
  <author>Yakushi~Kabuto</author>
  <version>20070321</version>
  <rule start_offset="10">
    <data offset="0" value="4E4553"/>
  </rule>
</detector>
This works great for 'Making a DAT' but fails with scanning because of a few files which have
NESM = 4E 45 53 4D in bytes 16-19
Code: [Select]
4E 45 53 1A 02 01 01 00 00 00 00 00 00 00 00 00
4E 45 53 4D 1A 01 01 01 80 80 80 80 84 80 53 6F
So I tried using yours:
Code: [Select]
<?xml version="1.0"?>
<detector>
  <name>Roman's NES Header Skipper</name>
  <author>Roman Scherzer</author>
  <version>1.0</version>
  <rule start_offset="10" end_offset="EOF" operation="none">
    <data offset="0" value="4E45531A" result="true"/>
    <data offset="8" value="0000000000000000" result="true"/>
  </rule>
</detector>
When using your "Header Skipper" to make a dat, I quickly discovered that 85 files I had were
datted as the 'whole file' = no bytes were skipped in the HASH calculation.
Some had 16 Byte - iNES 2.0 headers and the rest had 'dirty headers' with 'odd' bytes between 8-16
Examples
4E 45 53 1A 04 04 C0 30 00 01 00 00 00 00 00 00
4E 45 53 1A 04 04 80 C0 00 01 00 00 00 00 00 00
4E 45 53 1A 20 40 52 00 04 00 00 00 00 00 00 00
Anyway, I guess the easiest fix is to add the 1A
Code: [Select]
<?xml version="1.0"?>
<detector>
  <name>No-Intro NES Dat iNES Header Skipper</name>
  <rule start_offset="10">
    <data offset="0" value="4E45531A"/>
  </rule>
</detector>

This is new to me - so tell me is this line in your rule expecting bytes 8-16 to all be Zero ?
    <data offset="8" value="0000000000000000" result="true"/>
So if a header is polluted with DiskDude! your skipper will 'not ignore' the header when hashing ?

Oxy
Logged
Pages: [1] 2 3 4   Go Up
 

Page created in 0.16 seconds with 21 queries.

anything