Topic: Work In Progress (Read 51133 times)

Roman · « **Reply #40 on:** 12 August 2011, 21:23 »

ok...some fine progress this evening...

remember, I had to adjust all internal ziproutines to fully use the lib...fine...wonder why I didn't do it before since it's pretty easy....so say goodbye to strict scanning option, some internal zip repair etc...

Renaming files (or multiple files) within a zip is just some few lines of code now....so inplace rename, case fix, datetime fix are all done...

Still on the list (and that will take a bit more work I guess) is the rebuilder no-recompress option...but the lib got something for this I bet....

Will do some speedtest of heavy renaming in zips or reading zips to see how the lib performs...actually I think it will be faster.

oxyandy · « **Reply #41 on:** 13 August 2011, 04:33 »

I am happy to help speed test..

I have a set approx 1500 zips, which requires approx 20,000 renames.

Big enough to bench-mark the renaming with "Scanner"
then repeat with current release, if this helps.

Roman · « **Reply #42 on:** 13 August 2011, 20:02 »

one quick test...

a full MAME 143u2 scan (fullly diskcache buffered though): old and new scanner 18 seconds....maybe the old one a little bit faster (17seconds)

but hey, the old one was really optimized down to very limited reading of some fields only....

a full rename of MAME32 snap files in the zip (3688 times *.png -> *.xxx

): old cmpro 36 seconds...new one 10 seconds

yummi....

oxyandy · « **Reply #43 on:** 14 August 2011, 06:39 »

That really is a quick test..
Seems faster and faster is good.

I have done more testing with beta 2.
It really puts a smile on my face.
I was able to batch load a Tosec dat set, without having errors spit out at me everywhere..
I usually need to go into the dats manually and edit them to just get them to load.

Hanging out for next release.

oxyandy · « **Reply #44 on:** 14 August 2011, 11:31 »

I thought I was having some kind of hardware failure, possibly RAM.. Hi-end DDR3 4GB..
Is really hot where I live, so hardware failure isnt unlikely.
So, I ran Memtest-86 for a few cycles, no errors.

These betas must have some bad calls to ram ?

This sound right Roman, do you know of this ?

I have a folder with the beta and it's files, and it is constantly, but randomly, crashing.

Maybe I should send the folder 'as is' and some crash logs to you, to repeat the problem.

This seems to be common with all errors:
instruction at "0x0041d280"
So hope this means something to you.

Roman · « **Reply #45 on:** 16 August 2011, 12:47 »

I told you that your version is only for a quick test. It uses some not filled structures since I only hooked up a quick'n dirty lib based reader for you which only fills the main attributes for a utf8 name check. Other internally attributes were not correctly filled so that routines like zip in place rename, no-recompress copy and some others will fail....

oxyandy · « **Reply #46 on:** 17 August 2011, 03:14 »

I am not talking about the last build "new-reader"
the previous version..

Anyway, next build, I know exactly what eЯЯors to look for

Roman · « **Reply #47 on:** 17 August 2011, 20:24 »

Some news..

- completely replaced own zip routines with library. This includes inplace rename, copy-without-recompress (which is actually just a few lines with the lib...coool..hooked up scanner/rebuilder no-recompress to use this...)....some speedtests will follow....
- removed: rebuilderAdvanced buffer and auto mem options for recompress
- removed: compressor / zip settings / full zip structure scan

Next step....testing...and having a look at the new device_ref MAME stuff....

Update...
Little rebuilder no-recompress test....

2315 single files in 240 zips, total zip size is ~3GB (somehow reminds me of some famous bios based collection...

New cmpro: 4 min 45 secs
Old cmpro: 7 min 21 secs

yummi.....

oxyandy · « **Reply #48 on:** 19 August 2011, 04:01 »

unicode_test_20110818

Not much to report yet. Seems to be doing ok.

1. Scanner results window, copy set name to clipboard is only copying first letter.
However fix.dat is saving fine.

2. Batch mode - Rebuilding from 7z & Rar archives seems not to work. ( With either remove source ticked or not )
Zips fine.
Rebuilder, same.
I get the "Want to stop ?" Message. (oh this is with non-unicode file names, so nothing special)

Dragging -dropping either zip, 7z or rar, onto 'about' window, gives correct output. (with unicode file name & contents)
Dir2Dat - 1 folder 3 archives, zip, rar & 7z, gives correct output.

Scanner - With either zip, 7z or rar archived sets, reads fine.

Roman · « **Reply #49 on:** 19 August 2011, 07:05 »

sounds good! thanks for testing...
1 sounds like tiny bug which maybe related to some char* to TCHAR conversions...I will check that...should be an easy fix
2 sounds hmm interesting...especially since I've done some rar rebuilds lately....but I will check that too...

in the meanwhile I'm working on the "devices" stuff for new MAME....

Roman · « **Reply #50 on:** 19 August 2011, 21:50 »

ok...fixed the clipboard issue and added partly added "device" support

more tomorrow...rar/7z fix and finalize device support

Roman · « **Reply #51 on:** 21 August 2011, 20:09 »

so so so...coming closer to a release I guess (currently facing maybe last week in August)...
Thanks to the testers all reported issues are now history...

Additionally some preliminary devices / device_ref was added...well at least they are not handled as bios sets anymore and you can assign a path (like standard or mechanical) to them. There are some restrictions (like device != bios, only one device_ref) but time will tell what MAME will do and what cmpro should check.

On my list now is some final testing (e.g. with set-subfolders) and actually a new documentation....

Thanks again to all testers

oxyandy · « **Reply #52 on:** 22 August 2011, 05:03 »

I have tested every previous issue reported so far.

Yet to find a glitch,
nothing more said, means nothing found.
I am not done yet.

oxyandy · « **Reply #53 on:** 22 August 2011, 11:05 »

Bored with testing....

Finding a fault is too hard

Rebuilder
Sources
from rar
from 7zip & t7zip
from zip & tzip
from extracted files
Read Only Archives (such as found on DVD/CD)

Any type of output works.

Scanner
7z set
Rar set
Zip set
Read Only Sets

Batch Mode
No glitches loading any dat so far.
Destination folder, sets right path.
Rebuild, remove source, then Scan
Any output.

Dir2Dat
Extracted files with and without sub-folders
Zips with internal folders
Archives with Unicode Characters in file-names and contents

EDIT: OK, like the last time, I now have a replicable crash.
In my example,
after a batch run completes and CMP is showing the "Looking for dat files" msg.

The instruction at "0x0041d300" referenced memory at .......

When I have a chance,
I will transfer my CMP folder 'as is' to other pc or laptop to see if it repeats.

Roman · « **Reply #54 on:** 22 August 2011, 12:32 »

erm...so what are the exact steps for you issue again? With all these "working" things, I don't see what belongs to the error and what not...Please only list the issue again..thanks

ok..after rereading it's happening while the profiler reloads the profiles/datfiles after a completed batch run....interesting...trying to repeat it here somehow....if you can create a minimum setup....would be great.

oxyandy · « **Reply #55 on:** 22 August 2011, 15:54 »

The exact steps.

After successfully loading various runs of dats files in batch mode
and having them complete processing as expected.
The crashes start.
Well actually the crashes also happen at other random points.
So I should say, they become regular and predictable.

I have created a screen capture most times when it has crashed.

It doesn't happen when the program loads.

I think it is best you take my whole CMP folder 'as is'

HMM
The easiest and fastest way to repeat the crash, is to right click,
delete any entry in the MESS folder.
I do not feel any file in the MESS folder is the cause...

Anyway if the program crashes on your pc
the output will go straight to the 'debugger' -- right ?
Will this will give you a better idea... ?

Is about 50meg 7z..

Roman · « **Reply #56 on:** 22 August 2011, 16:11 »

well...screenshots of the windows error message won't help me...it would help me to tell me *where*/*when* it happens...
Right click and delete mess files? right click where? profiler? scanner?
I will do some research this evening...but if you can provide the files somewhere, that would be great...
"randomly" of course doesn't sound that good....maybe a not initialized variable...don't know...need to repeat it...then the debugger will catch it....

I've done several scans with and without rebuilder and never had that error....so it's not that common

So...a full dump of your folder would be a start...plus instructions how to repeat it...
...or maybe it only happens when scanning zip files....or only rar ... or only 7z files....? Try to minimize the scope...
...and of course turn off virusscanners...

oxyandy · « **Reply #57 on:** 23 August 2011, 02:13 »

Virus scanner off, yes.

Is not happening during any kind of actual processing.

It is happening After : when CMP shows the message "Looking for dat files" such as after a batch run,
however in the process of reducing the size of my CMP to share with you.
I found after deleting a dat (in this case from a profiler folder I have called MESS)
CMP then shows the message "Looking for dat files" and crashes.

This is the fastest way to repeat the crash.
This is happening 8 times of ten

It is not happening when I first open CMP, even though CMP shows the message "Looking for dat files"

It is not happening during a large batch operation, rebuild, match, remove then scan, then wait 1 sec, do next.
This all completes faultlessly, but then CMP after the batch run,
shows the message "Looking for dat files" and then crashes.

So when I say randomly crashing I never mean during a job.
So the type of archive 7z zip rar, has nothing to do with it.
I think with all the heavy testing I have done, these routines work really well.
(reading & writing archives, of any kind, no bugs here)
No bugs during the real 'work'...ok

I have emailed a link to my actual CMP folder 'as is'
To repeat the crash all you need to do is right click, delete an entry in the profiler, it will happen.
(here's hoping)
It wont matter that the paths etc don't exist, cause you don't even need to load a profile.

I am about to move over to another OS and try the folder 'as is' see if it repeats.
Will post results shortly.

oxyandy · « **Reply #58 on:** 23 August 2011, 02:27 »

Ok, really good news,
I extracted the archives contents here on this OS,
opened CMP, went to the MESS profile folder,
(however deleting ANY entry in any profiler folder should do the same crash)
delete a entry, crash
Repeated 5 times, every-time it crashes.

I already think I know what you will say is (part of) the cause,
but I know if you can stop it,
it will only make CMP tougher.. he he

Remember it has already loaded and successfully processed all the dats
that are currently in the profiler, very well.

Roman · « **Reply #59 on:** 23 August 2011, 06:59 »

Sounds like an array index access issue...however I wonder why it crashes for you after a batch run (which doesn't remove any profiles)...and actually this should also happen on the last official, non-unicode build.....anyway...I will do tests later today...and I guess I should check both, 64 and 32 bit

News:

Author Topic: Work In Progress (Read 51133 times)