EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: some updated build  (Read 17732 times)

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
some updated build
« on: 04 September 2012, 20:48 »
Well, some user told me that new alpha builds of 7z use lzma2 as default compression and cmpro is not able to handle them....so I had a quick look at the latest available 7z SDK, compiled cmpro with it and I'd say one 7z problem less....grab it here....

http://mamedev.emulab.it/clrmamepro/binaries/cmp20120904.rar
Logged


Starshadow

  • Member
  • *
  • Karma: 1
  • Offline Offline
  • Posts: 42
  • Operating System:
  • Linux Linux
  • Browser:
  • Firefox 14.0.1 Firefox 14.0.1
    • View Profile
Re: some updated build
« Reply #1 on: 05 September 2012, 14:35 »
Excellent! It's working great for me so far.

The new 7z alpha also supports renaming files inside 7z archives from the command line. Is there any chance of support for renaming files that way in cmpro if its supported by the user's copy of 7z.exe? It would be preferable to the current unarchive/rename/rearchive method.

Thanks.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Windows XP Windows XP
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
Re: some updated build
« Reply #2 on: 05 September 2012, 15:22 »
well...I'd like so see a good 7z SDK with easy and understandable blockwise compress/decompress 7z file (not LZMA streams) support without using COM ...but I guess that will never come...so we have to live with the exe files...

yes...if I find some time I will look at the rename operator (same for rar).
Logged

oxyandy

  • Member
  • *
  • Karma: 5
  • Offline Offline
  • Posts: 269
  • Operating System:
  • Windows XP Windows XP
  • Browser:
  • Firefox 15.0 Firefox 15.0
    • View Profile
    • .
Re: some updated build
« Reply #3 on: 05 September 2012, 15:43 »
Hey Roman..
Remember those "Please Dat Me" 7zips... ?
I was really hoping this build would help with those.
But...


Maybe next time ?
« Last Edit: 05 September 2012, 15:44 by oxyandy »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
Re: some updated build
« Reply #4 on: 05 September 2012, 18:53 »
Get more RAM or write a good library...

The SDK 7z C version extracts files completely to RAM...haven't found the time to check if this can be changed easily...but switching to the C++ 7z part with COM crap (which seems to support blockbased unpacking) ain't a way out if you ask me...
Logged

Cassiel

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 106
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Firefox 15.0 Firefox 15.0
    • View Profile
Re: some updated build
« Reply #5 on: 05 September 2012, 22:56 »
Quote from: Roman on 05 September 2012, 15:22
(same for rar).

That would be great...
Logged

oxyandy

  • Member
  • *
  • Karma: 5
  • Offline Offline
  • Posts: 269
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Firefox 15.0 Firefox 15.0
    • View Profile
    • .
Re: some updated build
« Reply #6 on: 06 September 2012, 02:52 »
Hi Roman....again,
Ok, with the "Please Dat Me" archive, the new build posted here..
I tried with 'win 7 x64' so using my full 4G of RAM
DIR2DAT - calculate SHA1 ticked
and I still get the same 7z MEM error.

Do you still have that archive, I know life is busy for you...
(I can up it again ?)
But just promise "one day" you will take a look...
With XP32, is already at system RAM max 3Gb so adding more is not an option.
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Mac OS X Mac OS X
  • Browser:
  • Safari 5.1 Safari 5.1
    • View Profile
Re: some updated build
« Reply #7 on: 06 September 2012, 05:47 »
please make the archive available again.
xp is not a state of the art system. I got 16gb on my win7 machine ;)

the problem is that the extract function in the 7z C core which doesnt seem to support blockwise unpacking. feel free to find me a better solution. But maybe I find some time to study the not-blockwise one to write my own....or switch to the C++ interface...yikes....why can't that sdk have a documentation like ZipArchiveLib....grrr :) (However I do respect the 7z work)....
« Last Edit: 06 September 2012, 07:07 by Roman »
Logged

oxyandy

  • Member
  • *
  • Karma: 5
  • Offline Offline
  • Posts: 269
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Firefox 15.0 Firefox 15.0
    • View Profile
    • .
Re: some updated build
« Reply #8 on: 06 September 2012, 11:21 »
Ok, you asked for it.
Here is another example of 7zip failure.
Using XP32 trying to Rebuild a merged 7z archive of acheart.7z direct from 0.146 u5 dat -
fails with an unspecified error.
(I know is old, but win 7 just cant handle video the way I need)

Anyway - here is the original Please Dat Me archive
http://speedy.sh/RBgFg/Please-DAT-Me.rar

DIR2DAT win7 x64 4Gb RAM - SHA1 ticked instant fail.
« Last Edit: 06 September 2012, 13:54 by oxyandy »
Logged

oxyandy

  • Member
  • *
  • Karma: 5
  • Offline Offline
  • Posts: 269
  • Operating System:
  • Windows XP Windows XP
  • Browser:
  • Firefox 15.0 Firefox 15.0
    • View Profile
    • .
Re: some updated build
« Reply #9 on: 06 September 2012, 14:40 »
Ok, while on subject of 7z.
Now let's say I have a single zip called Archive_1.zip
I also have it's matching DAT.
Inside the archive is 10 files 1.rom. 2.rom. 3.rom...... to 10.rom.

I load the dat, select the source folder which contains Archive_1.zip.
I select "Rebuilder" and Rebuild to New Folder with 7z as compression.

(Now let's see if I get this right)
CMP "Rebuilder" sees 1.rom matches the DAT and creates a 7z Archive_1.7z
with only 1 file in it.
It then sees the next file, 2.rom and now creates an Archive_1.7z
this time it creates an archive with 2 roms...
ETC..
Each time it finds a DAT match it only add 1 rom at a time.
So it makes Archive_1.7z 10 times !
This makes CMP Rebuilding to 7z painfully slow.
Why is this ?

Why doesn't "Rebuilder" see that Archive_1.zip contains all 10 matching
DAT entries the first time and then only make Archive_1.7z once ?
« Last Edit: 06 September 2012, 14:44 by oxyandy »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Windows XP Windows XP
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
Re: some updated build
« Reply #10 on: 06 September 2012, 15:15 »
Rebuilder is file based, not set based. It creates each instance of matched file X in the source in the destination sets Y1, Y2,...Yn. Then it continues with the next file in the source (no matter if within an archive or not). That's the way it is...and for zip it doesn't really matter (especially if you got no recompression on) since it can easily move to the correct place to add the stuff....7z solid archives are of course a pain.

Maybe in the far away future I add some lookahead or caching thing...but don't count on it.
« Last Edit: 06 September 2012, 15:23 by Roman »
Logged

oxyandy

  • Member
  • *
  • Karma: 5
  • Offline Offline
  • Posts: 269
  • Operating System:
  • Windows XP Windows XP
  • Browser:
  • Firefox 15.0 Firefox 15.0
    • View Profile
    • .
Re: some updated build
« Reply #11 on: 06 September 2012, 15:45 »
Yes, I see,
I was thinking that if CMP "Rebuilder" somehow "Looked and Remembered".....
What was where, what matched and what needed to go where...
A "First Pass".
Store that info in some sort of buffer file, 'before' it actually created a single archive.

Then once it had completed the "First Pass"
It went about "Rebuilding" the files based on the "Buffer File" contents..
Would the process be ultimately faster ?
But, then I think about how this completely rewrites the way CMP currently works,
the complexity required... hmmm
And decide is best to keep my thoughts to myself... hehe
Especially considering I see <---- You have XP loaded..

Thanks again for the time and effort you put into CMP.
It's priceless !
Cheers
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
Re: some updated build
« Reply #12 on: 06 September 2012, 18:29 »
well, I got dozens of ideas for a new scanner and rebuilder...but I simply don't have time...

In times of terabyte hds compression advantages of 7z vs zip doesn't really play a role....but I'm still happy if I can improve it here and there a bit....

Regarding the rebuilder...maybe I find some time in my parental leave time....time will tell...


...and by the way...The "Please Dat Me" test succeeded without sha1 within a second, with SHA1 it took way longer of course and mem usage of my system went up to 5.39GB maximum during processing the 2nd 7z (with cmpro, browser, email etc running)....first file it not even reached nearly 3GB.....
« Last Edit: 06 September 2012, 18:47 by Roman »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Windows XP Windows XP
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
Re: some updated build
« Reply #13 on: 07 September 2012, 12:42 »
now you made me horny writing a new rebuilder core..... :O)
Logged

oxyandy

  • Member
  • *
  • Karma: 5
  • Offline Offline
  • Posts: 269
  • Operating System:
  • Windows XP Windows XP
  • Browser:
  • Firefox 15.0 Firefox 15.0
    • View Profile
    • .
Re: some updated build
« Reply #14 on: 07 September 2012, 15:11 »
 ;D
Really ?
Ok well, please share some of your thoughts and progress.
Problems and successes.
And last of all, keep the Mrs happy.
Oh and any beta builds..


Logged

sirvh

  • Karma: 0
  • Offline Offline
  • Posts: 7
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
Re: some updated build
« Reply #15 on: 22 September 2012, 17:05 »
Quote from: Roman on 07 September 2012, 12:42
now you made me horny writing a new rebuilder core..... :O)

Something for the future but, do you plan to make something like Romvault does? I mean scan a folder and detect from which dat the file belong and rebuild to the proper folder?
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Offline Offline
  • Posts: 3445
  • Operating System:
  • Windows 7/Server 2008 R2 Windows 7/Server 2008 R2
  • Browser:
  • Chrome 21.0.1180.89 Chrome 21.0.1180.89
    • View Profile
Re: some updated build
« Reply #16 on: 22 September 2012, 17:26 »
use the cmpro batcher to run a rebuild-only over all your dats and chosen folders....will do the same...
Logged
Pages: [1]   Go Up
« previous next »
 

Page created in 0.121 seconds with 20 queries.

I Cookies ci aiutano a fornirti una migliore esperienza di navigazione. Utilizzando il nostro sito, ci autorizzi all'uso dei cookies.