Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[B2K24]

We have been able to consistently reproduce everytime  cmpro 4.08b deletes or loses some roms/sets for unknown reason when scanning + fix a 147 complete romset with a self compiled 147u2 or 147u3 MAME binary.

FYI since we are working with merge sets here at the Datfile problem prompt we answer Yes To All about removing parent/clone relationship.

This happens when working with a complete 147 merged/zip set and also same result happens with merged/7-zip

strange thing is there is a different amount of roms that cmpro moves to backup path depending if your doing merged/zip or merged/7-zip

Here are some quotes from a friend testing along with me.

I was able to reproduce the problem.

While updating my 0.147 7z/merged set to 0.147u2, I ended up missing 20 ROMs.
These were not part of the 0.147 to 0.147u3 update torrent.
They were not part of my CMP backup folder either.
However, they were part of the 0.147 7z/merged set I started with (I made a backup of that)!
The ROMs must have been deleted from my set while scanning/fixing the 0.147 set with the 0.147u2 datfile.
That leaves us with the following questions:

It's a CMP problem, but is it 7-zip related?
Is it a problem CMP deleting ROMs while editing 7-zip files instead of moving them to the backup folder?
If it is, does CMP use an internal 7z routine to scan/fix or does it use the 7-zip version installed on the system and assigned in the CMP settings?
If it's external, which 7-zip version can be used to avoid this problem (9.22 used)?
Unlikely option: is it perhaps the t7zip file CMP (7z) has problems with?

To be continued...

Interesting.
This strikes a few questions.
The same happens with a merged/zip set.
CMP deleted ROMs when scanning/fixing the 0.147 set with the 0.147u2 datfile.
This time more ROMs ended up in the backup folder, 10 ROMs were deleted.
All 10 missing ROMs were part of the original 0.147 ROMs set I started with.

To be continued...

Will try to get more information as we figure things out. Hopefully Roman or Oxy can understand what is taking place here.

[oxyandy]

It's external... 9.22 doesn't support renames
Use 9.28a - re-do your test



Bye

[oddi]

@oxyandy: Your mame sets 7z/merged ? When u scan sets any problem with lost old roms ?
btw -now used 9.28a ( wait few mame svn updates for test), with 9.29 and 9.30 have problems.

[Dullaron]

I lost some ROM's on merging the MAME ROM's set. Only way actually merge the sets is to edit the Split ROM's dat and then merge the right way. There is a reason to not merge with the clrmamepro without having a good merge dat. The warning on the clrmamepro screen is saying not to do it. A lot of people think it OK to merge but they don't know the ROM's will replace others. If you don't know what your doing then it to be safe to use Split until you find a good merge dat that is been made by a human not clrmamepro.

Example: When there a ROM in a main set or clone set. One of those ROM is going to replace the other ROM with the same name and different crc. You don't want to merge those.

No problem merge MESS ROM's though. No warning screen popping up. MESS seem clean and safe to merge.

Oh I'm using 7-Zip 9.20 (2010-11-18). I will try the beta.

[oxyandy]

@oddi
I have for the longest time had my sets merged and t7z'ed...
MAME only recently, but always merged.
The full MESS softlist all merged..
and yes problems with post 9.28+ versions of 7z,
(so the version of 7z is only an intermediate step - one that 'works' does fine)
because I always 'Send To' my whole MESS root folder to t7z after updates,
hence the settings.. (these settings are t7z friendly)

I find maintaining sets this way super easy..

@Dullaron
I use default dats and never lost a ROM..
I think your techniques  cause ROM loss..
For example:
When updating, never rebuild files directly into a set with doing a "Scan" first..

[B2K24]

@oxyandy

I have installed 9.28a and set my 7z tab in compressor properties identical to yours in your pic. The problem with roms missing still can be 100% reproduced and it seems to be the exact same roms/sets everytime!
Let me explain the steps I use to reproduce this issue.

- load official MAME 147 binary into CMP
- rebuild complete 147 romset into empty destination directory with merge sets ticked and zip or 7z does not matter
- after rebuild is complete jump to scanner and audit destination directory which shows 0 misses
- right click in the profiler and delete profile + clean cache
- load self compiled 147u3 into profiler
- scanner execute scan + fix all option with you prefer merge sets
- rebuild from source MAME - Update ROMs (v0.147 to v0.147u3) with merge sets ticked
- rebuild from backupPath

result is 27 are missing so where did they go?

http://pastebin.com/mxNBmuh0

[Dullaron]

Would be a good idea having the set up to where if a ROM matches the name and having different CRC then don't merge over to the main set. That ROM stay in that clone set. No complains ever. Well this is my last post on this.

See these. Won't be added on Merge. 21 will be miss. From the svn.

Code: [Select]

Set: Best League (bootleg of Big Striker, Italian Serie A)
Name: bestleag
ROM: 2.bin

================

Set: Dynamite Duke (Japan)
Name: dyndukej
ROM: 3.e8

================

Set: Trivia (Unsorted question roms)
Name: gt103a1
ROM: nfl_football

================

Set: Trivia (Version 1.03a Sex questions)
Name: gt103asx
ROM: adult_sex_4

================

Set: Invader's Revenge (set 2)
Name: invrvngea
ROM: e.ic33

================

Set: Invader's Revenge (Dutchford, single PCB)
Name: invrvngedu
ROM: ir.1t

================

Set: Ms. Pac-Man Heart Burn
Name: msheartb
ROM: 5e

================

Set: Pleiads (bootleg set 1)
Name: pleiadbl
ROM: ic47.bin

================

Set: Pleiads (Centuri)
Name: pleiadce
ROM: ic47.bin

================

Set: Pleiads (bootleg set 2)
Name: pleiadsb2
ROM: ic47.bin

================

Set: Pool 10 (Italian, set 4)
Name: pool10d
ROM: 1.u20

================

Set: Pool 10 (Italian, Dino 4 hardware, encrypted)
Name: pool10e
ROM: 1.u20

================

Set: Raiden Fighters Jet (Asia)
Name: rfjeta
ROM: prg1.u0212

================

Set: Raiden Fighters Jet (Taiwan)
Name: rfjett
ROM: prg0.u0211

================

Set: Rough Ranger (v2.0, unprotected, bootleg?)
Name: rranger
ROM: 1

================

Set: Rush & Crash (Japan)
Name: rushcrsh
ROM: rc01.11e

================

Set: The Speed Rumbler (set 2)
Name: srumbler2
ROM: rc01.11e

================

Set: The Speed Rumbler (set 3)
Name: srumbler3
ROM: rc01.11e

================

Set: Super ABC (Pac-Man multigame kit, Mar. 08 1999)
Name: superabco
ROM: superabc.u14

================

Set: WEC Le Mans 24 (set 2)
Name: wecleman2
ROM: 602f08.17h

================

Set: Mahjong Yoshimoto Gekijou [BET] (Japan)
Name: yosimotm
ROM: 1.7c
I don't why they didn't put alt on the ROM's when there different crc.

oxyandy I going try the dat then.

[oxyandy]

@Dullaron

I don't why they didn't put alt on the ROM's when there different crc.

I so agree with you on this !
I also asked Roman once if CMP could do the same...(add the _alt)

@B2K24
I used the 0.147 "ROMs only dat from ARMAX" merged set, zero miss...

Then I used the 0.147 u3 "ROMs only dat from ARMAX"
Scanner, with all fix options ticked...
I did not need to go further.

The result ? CMP apparently ate ROMs that were in 0.147..
(But the ROMs are there, just CMP is showing them incorrectly as missing)

Looking at just one.. (ds_sh0.rom)
Appears 3 times in 0.147_dat
Appears just once in 0.147u3_dat

<rom name="ds_sh0.rom" merge="a3-sh0-c-0.ic31" size="65536" crc="23fe6ffc" sha1="896377961cafc19e44d9d889f9fbfdbaedd556da"/>

Guess the name change like this is causing issues ?
ds_sh0.rom   to   a3-sh0-c-0.ic31

[oxyandy]

Hi Roman,
Here is two mini dats and an archive to reproduce the problem
http://speedy.sh/PFmqn/TEST-MAME-ROMs-v0.147-v0.147u3.zip

1. TEST_MAME - ROMs (v0.147_XML).dat
Get this set complete. Passes Audit.

2. TEST_MAME - ROMs (v0.147u3_XML).dat
Load this dat, point to path of above set.. Scan only with fix options ticked.

Scanner result showing missing ROMs (Merged 7z or Zip)
(But the needed CRC files exist in the archive)

Enjoy 

Edit: I should add the sets are copied/pasted directly from the parent dats..
Fault found in Merged Mode.

[oxyandy]

Actually the ROMs are not missing at all..
If anyone does my test 'as above' ...
Then once you go get the u3 Scanner result saying 5 ROMs are missing..
Go look in the archive, you will see  all the CRCs are there...

So the MAME exe audit would not see anything missing...
0.147 expects 29 files (merged in the archive dsoccr94.zip)
0.147u3 expects 34 the difference being dupes 

[Dullaron]

You seeing the problem now. This is the way MAME doing this. Not clrmamepro. clrmamepro only seeing on what MAME telling it. I'm sticking to Split. MESS on the other hand is clean.

[Roman]

long thread...and actually I just (22:20) came back from a weekend trip....and after a long drive I'm too tired to read the full thread....
actually I will check oxyandy's "Hi Roman,Here is two mini dats and an archive..." setup in the next days to see what's going on...

Before checking, of course people have to be aware that:
- deleted files from 7z archives are moved to backup but in a zip archive
- there are some cases (depends on the parent/clone changes from one dat to another and the order of the sets), where you need to rebuild files from backup to get them back. This is intended.
- there are some cases where a scan shows missing sets and you simply have to do a fastscan to get the clear result.
- if you don't care about dupes you need to use profiler's option to parse rom merge tags
- if you do care about fake clones (fully identical sets), there's also a setting there...

It's absolutely needed to send me scenario where I can rebuild issues. So a minimum combination of cmpro (with settings, cmpro.ini, datfile) and a minimum set of roms. I want to press a button and see the problem.

[Dullaron]

OK I guest I'm wrong about the merge sets. Tested MAME 0.147u3. Merge first and then Split on the Jump to Rebuilder. All I did is switch folders for a test. Only miss 4 because I roll back to released 0.147u3. Newer svn source I wasn't using this time. No ROM's was missing as oxyandy said. But yes rebuild the whole sets will solve the problem on merge. Must make a new folder for rebuild.

Roman is right. Gotta do clean rebuild the sets. Problem solve for me.

[Roman]

So since people tell me that there is a post on pleasure dome form (which I cannot access) on this...

Can anyone clearly say there is or there is not a problem, and if there is, can send me a scenario for it?

[oxyandy]

Roman,
The fault was first explained by B2K24..
(But that is a lot to get through)

So I simplified it..
With my 2 mini dats...
(Use these as I describe, you will see what they mean on PD)

However, let me explain it !

Without  "Parge ROM Merge Tags"  ticked CMP will report 5 missing ROMs,
on a simulated update from 0.147 to 0.147u3.. (my mini.dats)
With a Merged Set, either ZIP or 7z.. (When in fact, the files are in the archive)

However, if the archive's 'so called missing' files are found from a different source,
CMP will Rebuild them in, resulting in an Archive with 34 internal files, (incorrect)
now passing Audit in CMP.

With  "Parge ROM Merge Tags"  ticked will result in an Archive with 29 files (correct)

So, of course, you are completely correct...
Using my 2 mini dats, with "Parge ROM Merge Tags" ticked in the profiler..
The 'so called problem' is gone...

Maybe "Parge ROM Merge Tags" should be 'enabled by default'



Cheers
Oxyandy

[oddi]

@oxyandy: This option if enable solved problem ?
...testing progress ..

[EDIT]
After this option ticked, give example with 1 rom from 1941 set:

3a00dd5772f38081fde11d8d61ba467379e2a636 *41_09.12b
3a00dd5772f38081fde11d8d61ba467379e2a636 *41_23.13b

maybe that is solution, enabled

[oxyandy]

Maybe that is solution, enabled

Do not sound so uncertain, do some testing for yourself..
Come back change the word 'Maybe'
 

[Roman]

well, I'm currently at work so I can't check it for now...
regarding "parse rom merge options"...It is disabled because:

- the merge tags in the (MAME) dats were too often too wrong in the past (pointing to wrong roms, holding wrong checksum information and so on). Actually during cleaning up the datfile I've added verification tests in the past to fix such bad merge tags automatically....

- I'm more a fan of having the correct number of files in the sets with the correct naming....even when fully merging the sets you can differ the clone names then..(so you got multiple files with the same checksum and different names). In days of multi terabyte hds and users who do collect mame roms, space shouldn't be an argument here...I still wonder why people switch to 7z because of this...


what I wonder about:

"Without  "Parge ROM Merge Tags"  ticked CMP will report 5 missing ROMs,
on a simulated update from 0.147 to 0.147u3.. (my mini.dats)
With a Merged Set, either ZIP or 7z.. (When in fact, the files are in the archive)"

Where does it report them? In the stats? Well...the only correct values are the scan tree view. Don't care about the stats at the end...I never claimed they are correct...since they rely on so many factors. If nothing is in the treeview you're done.

[oxyandy]

what I wonder about:
Where does it report them? In the stats? Well...the only correct values are the scan tree view. Don't care about the stats at the end...I never claimed they are correct...since they rely on so many factors. If nothing is in the treeview you're done.

The answer to the above...
It reports them as Missing... like ordinary missing files...

[Roman]

If you're coming from a set where you had "parse merge tags" enabled (and so you did not need them) and you're now scanning them with a disabled "parse merge tags", they are of course missing...because this mode is strict...different rom names result in different rom files.

However, fix-missing (or rebuilder) should find them and create them automatically for you... If that's the scenario I don't see anything wrong (or even something like "cmpro ate my roms"...

but I will check your scenario tonight...