EMULAB Forum
clrmamepro [English] => clrmame Discussion => Topic started by: B2K24 on 02 December 2012, 01:33
-
We have been able to consistently reproduce everytime cmpro 4.08b deletes or loses some roms/sets for unknown reason when scanning + fix a 147 complete romset with a self compiled 147u2 or 147u3 MAME binary.
FYI since we are working with merge sets here at the Datfile problem prompt we answer Yes To All about removing parent/clone relationship.
This happens when working with a complete 147 merged/zip set and also same result happens with merged/7-zip
strange thing is there is a different amount of roms that cmpro moves to backup path depending if your doing merged/zip or merged/7-zip
Here are some quotes from a friend testing along with me.
I was able to reproduce the problem.
While updating my 0.147 7z/merged set to 0.147u2, I ended up missing 20 ROMs.
These were not part of the 0.147 to 0.147u3 update torrent.
They were not part of my CMP backup folder either.
However, they were part of the 0.147 7z/merged set I started with (I made a backup of that)!
The ROMs must have been deleted from my set while scanning/fixing the 0.147 set with the 0.147u2 datfile.
That leaves us with the following questions:
It's a CMP problem, but is it 7-zip related?
Is it a problem CMP deleting ROMs while editing 7-zip files instead of moving them to the backup folder?
If it is, does CMP use an internal 7z routine to scan/fix or does it use the 7-zip version installed on the system and assigned in the CMP settings?
If it's external, which 7-zip version can be used to avoid this problem (9.22 used)?
Unlikely option: is it perhaps the t7zip file CMP (7z) has problems with?
To be continued...
Interesting.
This strikes a few questions.
The same happens with a merged/zip set.
CMP deleted ROMs when scanning/fixing the 0.147 set with the 0.147u2 datfile.
This time more ROMs ended up in the backup folder, 10 ROMs were deleted.
All 10 missing ROMs were part of the original 0.147 ROMs set I started with.
To be continued...
Will try to get more information as we figure things out. Hopefully Roman or Oxy can understand what is taking place here.
-
It's external... 9.22 doesn't support renames
Use 9.28a - re-do your test
(http://ndsxdelta.no-ip.org/t7z_friendly.png)
Bye
-
@oxyandy: Your mame sets 7z/merged ? When u scan sets any problem with lost old roms ?
btw -now used 9.28a ( wait few mame svn updates for test), with 9.29 and 9.30 have problems.
-
I lost some ROM's on merging the MAME ROM's set. Only way actually merge the sets is to edit the Split ROM's dat and then merge the right way. There is a reason to not merge with the clrmamepro without having a good merge dat. The warning on the clrmamepro screen is saying not to do it. A lot of people think it OK to merge but they don't know the ROM's will replace others. If you don't know what your doing then it to be safe to use Split until you find a good merge dat that is been made by a human not clrmamepro.
Example: When there a ROM in a main set or clone set. One of those ROM is going to replace the other ROM with the same name and different crc. You don't want to merge those.
No problem merge MESS ROM's though. No warning screen popping up. MESS seem clean and safe to merge.
Oh I'm using 7-Zip 9.20 (2010-11-18). I will try the beta.
-
@oddi
I have for the longest time had my sets merged and t7z'ed...
MAME only recently, but always merged.
The full MESS softlist all merged..
and yes problems with post 9.28+ versions of 7z,
(so the version of 7z is only an intermediate step - one that 'works' does fine)
because I always 'Send To' my whole MESS root folder to t7z after updates,
hence the settings.. (these settings are t7z friendly)
I find maintaining sets this way super easy..
@Dullaron
I use default dats and never lost a ROM..
I think your techniques cause ROM loss..
For example:
When updating, never rebuild files directly into a set with doing a "Scan" first..
-
@oxyandy
I have installed 9.28a and set my 7z tab in compressor properties identical to yours in your pic. The problem with roms missing still can be 100% reproduced and it seems to be the exact same roms/sets everytime!
Let me explain the steps I use to reproduce this issue.
- load official MAME 147 binary into CMP
- rebuild complete 147 romset into empty destination directory with merge sets ticked and zip or 7z does not matter
- after rebuild is complete jump to scanner and audit destination directory which shows 0 misses
- right click in the profiler and delete profile + clean cache
- load self compiled 147u3 into profiler
- scanner execute scan + fix all option with you prefer merge sets
- rebuild from source MAME - Update ROMs (v0.147 to v0.147u3) with merge sets ticked
- rebuild from backupPath
result is 27 are missing so where did they go?
http://pastebin.com/mxNBmuh0 (http://pastebin.com/mxNBmuh0)
-
Would be a good idea having the set up to where if a ROM matches the name and having different CRC then don't merge over to the main set. That ROM stay in that clone set. No complains ever. Well this is my last post on this.
See these. Won't be added on Merge. 21 will be miss. From the svn.
Set: Best League (bootleg of Big Striker, Italian Serie A)
Name: bestleag
ROM: 2.bin
================
Set: Dynamite Duke (Japan)
Name: dyndukej
ROM: 3.e8
================
Set: Trivia (Unsorted question roms)
Name: gt103a1
ROM: nfl_football
================
Set: Trivia (Version 1.03a Sex questions)
Name: gt103asx
ROM: adult_sex_4
================
Set: Invader's Revenge (set 2)
Name: invrvngea
ROM: e.ic33
================
Set: Invader's Revenge (Dutchford, single PCB)
Name: invrvngedu
ROM: ir.1t
================
Set: Ms. Pac-Man Heart Burn
Name: msheartb
ROM: 5e
================
Set: Pleiads (bootleg set 1)
Name: pleiadbl
ROM: ic47.bin
================
Set: Pleiads (Centuri)
Name: pleiadce
ROM: ic47.bin
================
Set: Pleiads (bootleg set 2)
Name: pleiadsb2
ROM: ic47.bin
================
Set: Pool 10 (Italian, set 4)
Name: pool10d
ROM: 1.u20
================
Set: Pool 10 (Italian, Dino 4 hardware, encrypted)
Name: pool10e
ROM: 1.u20
================
Set: Raiden Fighters Jet (Asia)
Name: rfjeta
ROM: prg1.u0212
================
Set: Raiden Fighters Jet (Taiwan)
Name: rfjett
ROM: prg0.u0211
================
Set: Rough Ranger (v2.0, unprotected, bootleg?)
Name: rranger
ROM: 1
================
Set: Rush & Crash (Japan)
Name: rushcrsh
ROM: rc01.11e
================
Set: The Speed Rumbler (set 2)
Name: srumbler2
ROM: rc01.11e
================
Set: The Speed Rumbler (set 3)
Name: srumbler3
ROM: rc01.11e
================
Set: Super ABC (Pac-Man multigame kit, Mar. 08 1999)
Name: superabco
ROM: superabc.u14
================
Set: WEC Le Mans 24 (set 2)
Name: wecleman2
ROM: 602f08.17h
================
Set: Mahjong Yoshimoto Gekijou [BET] (Japan)
Name: yosimotm
ROM: 1.7c
I don't why they didn't put alt on the ROM's when there different crc.
oxyandy I going try the dat then.
-
@Dullaron
I don't why they didn't put alt on the ROM's when there different crc.
I so agree with you on this !
I also asked Roman once if CMP could do the same...(add the _alt)
@B2K24
I used the 0.147 "ROMs only dat from ARMAX" merged set, zero miss...
Then I used the 0.147 u3 "ROMs only dat from ARMAX"
Scanner, with all fix options ticked...
I did not need to go further.
The result ? CMP apparently ate ROMs that were in 0.147..
(But the ROMs are there, just CMP is showing them incorrectly as missing)
Looking at just one.. (ds_sh0.rom)
Appears 3 times in 0.147_dat
Appears just once in 0.147u3_dat
<rom name="ds_sh0.rom" merge="a3-sh0-c-0.ic31" size="65536" crc="23fe6ffc" sha1="896377961cafc19e44d9d889f9fbfdbaedd556da"/>
Guess the name change like this is causing issues ?
ds_sh0.rom to a3-sh0-c-0.ic31
-
Hi Roman,
Here is two mini dats and an archive to reproduce the problem
http://speedy.sh/PFmqn/TEST-MAME-ROMs-v0.147-v0.147u3.zip (http://speedy.sh/PFmqn/TEST-MAME-ROMs-v0.147-v0.147u3.zip)
1. TEST_MAME - ROMs (v0.147_XML).dat
Get this set complete. Passes Audit.
2. TEST_MAME - ROMs (v0.147u3_XML).dat
Load this dat, point to path of above set.. Scan only with fix options ticked.
Scanner result showing missing ROMs (Merged 7z or Zip)
(But the needed CRC files exist in the archive)
Enjoy :P
Edit: I should add the sets are copied/pasted directly from the parent dats..
Fault found in Merged Mode.
-
Actually the ROMs are not missing at all..
If anyone does my test 'as above' ...
Then once you go get the u3 Scanner result saying 5 ROMs are missing..
Go look in the archive, you will see all the CRCs are there...
So the MAME exe audit would not see anything missing...
0.147 expects 29 files (merged in the archive dsoccr94.zip)
0.147u3 expects 34 the difference being dupes ::)
-
You seeing the problem now. This is the way MAME doing this. Not clrmamepro. clrmamepro only seeing on what MAME telling it. I'm sticking to Split. MESS on the other hand is clean.
-
long thread...and actually I just (22:20) came back from a weekend trip....and after a long drive I'm too tired to read the full thread....
actually I will check oxyandy's "Hi Roman,Here is two mini dats and an archive..." setup in the next days to see what's going on...
Before checking, of course people have to be aware that:
- deleted files from 7z archives are moved to backup but in a zip archive
- there are some cases (depends on the parent/clone changes from one dat to another and the order of the sets), where you need to rebuild files from backup to get them back. This is intended.
- there are some cases where a scan shows missing sets and you simply have to do a fastscan to get the clear result.
- if you don't care about dupes you need to use profiler's option to parse rom merge tags
- if you do care about fake clones (fully identical sets), there's also a setting there...
It's absolutely needed to send me scenario where I can rebuild issues. So a minimum combination of cmpro (with settings, cmpro.ini, datfile) and a minimum set of roms. I want to press a button and see the problem.
-
OK I guest I'm wrong about the merge sets. Tested MAME 0.147u3. Merge first and then Split on the Jump to Rebuilder. All I did is switch folders for a test. Only miss 4 because I roll back to released 0.147u3. Newer svn source I wasn't using this time. No ROM's was missing as oxyandy said. But yes rebuild the whole sets will solve the problem on merge. Must make a new folder for rebuild.
Roman is right. Gotta do clean rebuild the sets. Problem solve for me.
-
So since people tell me that there is a post on pleasure dome form (which I cannot access) on this...
Can anyone clearly say there is or there is not a problem, and if there is, can send me a scenario for it?
-
Roman,
The fault was first explained by B2K24..
(But that is a lot to get through)
So I simplified it..
With my 2 mini dats...
(Use these as I describe, you will see what they mean on PD)
However, let me explain it !
Without "Parge ROM Merge Tags" ticked CMP will report 5 missing ROMs,
on a simulated update from 0.147 to 0.147u3.. (my mini.dats)
With a Merged Set, either ZIP or 7z.. (When in fact, the files are in the archive)
However, if the archive's 'so called missing' files are found from a different source,
CMP will Rebuild them in, resulting in an Archive with 34 internal files, (incorrect)
now passing Audit in CMP.
With "Parge ROM Merge Tags" ticked will result in an Archive with 29 files (correct)
So, of course, you are completely correct...
Using my 2 mini dats, with "Parge ROM Merge Tags" ticked in the profiler..
The 'so called problem' is gone...
Maybe "Parge ROM Merge Tags" should be 'enabled by default' ???
(http://ndsxdelta.no-ip.org/Parse_merge_tag.png)
Cheers
Oxyandy
-
@oxyandy: This option if enable solved problem ?
...testing progress ..
[EDIT]
After this option ticked, give example with 1 rom from 1941 set:
3a00dd5772f38081fde11d8d61ba467379e2a636 *41_09.12b
3a00dd5772f38081fde11d8d61ba467379e2a636 *41_23.13b
maybe that is solution, enabled :)
-
Maybe that is solution, enabled :)
Do not sound so uncertain, do some testing for yourself..
Come back change the word 'Maybe'
;D
-
well, I'm currently at work so I can't check it for now...
regarding "parse rom merge options"...It is disabled because:
- the merge tags in the (MAME) dats were too often too wrong in the past (pointing to wrong roms, holding wrong checksum information and so on). Actually during cleaning up the datfile I've added verification tests in the past to fix such bad merge tags automatically....
- I'm more a fan of having the correct number of files in the sets with the correct naming....even when fully merging the sets you can differ the clone names then..(so you got multiple files with the same checksum and different names). In days of multi terabyte hds and users who do collect mame roms, space shouldn't be an argument here...I still wonder why people switch to 7z because of this...
what I wonder about:
"Without "Parge ROM Merge Tags" ticked CMP will report 5 missing ROMs,
on a simulated update from 0.147 to 0.147u3.. (my mini.dats)
With a Merged Set, either ZIP or 7z.. (When in fact, the files are in the archive)"
Where does it report them? In the stats? Well...the only correct values are the scan tree view. Don't care about the stats at the end...I never claimed they are correct...since they rely on so many factors. If nothing is in the treeview you're done.
-
what I wonder about:
Where does it report them? In the stats? Well...the only correct values are the scan tree view. Don't care about the stats at the end...I never claimed they are correct...since they rely on so many factors. If nothing is in the treeview you're done.
The answer to the above...
It reports them as Missing... like ordinary missing files...
(http://ndsxdelta.no-ip.org/like_this.png)
-
If you're coming from a set where you had "parse merge tags" enabled (and so you did not need them) and you're now scanning them with a disabled "parse merge tags", they are of course missing...because this mode is strict...different rom names result in different rom files.
However, fix-missing (or rebuilder) should find them and create them automatically for you... If that's the scenario I don't see anything wrong (or even something like "cmpro ate my roms"...
but I will check your scenario tonight...
-
FYI Parse ROM merge tags option has always been unchecked in my clrmame since it's the default option.
-
so I did a little bit of research...
First of all, there is no file loss at all...as oxyandy said already, the files are in the archive but are listed as missing in the tree view...which is correct for this check, since the files do not exist with that name (rebuilder will of course create them since they are dupes and do exist with a different name).
The reason why you're stuck is that the fix missing (or test for missing-but-fixable) routine did not look into the parent set in case of full merged sets. It did however look into the parent set in addpaths or backup folder but not directly in the same set in the rompath...
You might want to try this build which includes this lookup (and a little goodie for oxyandy)
http://mamedev.emulab.it/clrmamepro/binaries/cmpro20121203.rar (http://mamedev.emulab.it/clrmamepro/binaries/cmpro20121203.rar)
-
i going to try that one. Before I do. I want clear this with oxyandy. I didn't do a clean rebuild this time and see what happen here. This is from the svn that I download this morning. Merge the ROM's setting.
Force 10 (JPM) (IMPACT) (set 5) [folder: j6frc10d - parent: j6frc10 - size: 768kb]
wrong size: f1015p2 [wrong: 131072] [right: 131073] [CRC32: 36508a26] [not fixed]
Happy Hour (JPM) (IMPACT) (set 1) [folder: j6hapyhr - size: 768kb]
wrong size: 20427.bin [wrong: 131072] [right: 131073] [CRC32: 3de94b07] [not fixed]
No matter what I do. This will show without doing clean rebuild. Will not removed those because of the same name and different crc? Looking at the same ROM and confused? Unless it swapping in and out with two different sets.
Anyway I took those out and then rebuild those sets. Now fix. Just wouldn't removed those for some reason.
Roman. The new tool did some fixing. I saw it doing some fixing. Just wouldn't fix those above.
-
look at the size. Your file is 131072 and it should be 131073....
how should cmpro guess the missing byte?
if "run checksum analysis" (and "fix if possible") is enabled (chd + checksum options window), it tries to pad with 00, 01 and 0xff if I remember correctly...and checks the checksum after that....but generally cmpro can't guess missing bytes....
131073 is however a very odd (ha ha ha) file size...maybe a bug in the dat.
but anyway...give the checksum analysis a try...
-
Thank You Roman, after using the version you linked a rebuild to 147 and then a scan + fix of 147u3 + rebuild from update pack and backup path gets 0 misses
Missing
·Sets 0/26452
·ROMs 0/205995
·CHDs -/623
·Samples -/3507
·Bytes 0/148gb
-
Roman I'm talking about the tool doesn't removed the useless ROM's when all the boxes are checked on those two. Everything else useless gotten removed without a problem.
ffight -> useless ROM removed. s224b.1a
qtono2j -> useless ROM removed. tn2j_09.12a
sc4gcclb -> useless ROM's removed. 95004252.bin 95004253.bin
Unneeded is checked.
I did removed the useless ROM's by drop in the window. I think the ROM's must've been fix by MAME Team or who ever. Probably missing header or something.
Anyway I wouldn't worry about it Roman.
-
Dullaron
If you look at the dat entries for these
Force 10 (JPM) (IMPACT) (set 5) [folder: j6frc10d - parent: j6frc10 - size: 768kb]
wrong size: f1015p2 [wrong: 131072] [right: 131073] [CRC32: 36508a26] [not fixed]
Happy Hour (JPM) (IMPACT) (set 1) [folder: j6hapyhr - size: 768kb]
wrong size: 20427.bin [wrong: 131072] [right: 131073] [CRC32: 3de94b07] [not fixed]From the last dat to the latest, you will probably notice that the CRCs are the same in each dat.. Only the 'apparent' size has changed...
If the above is true, then CMP is not considering the ROMs useless..
It is considering them as 'needed' ! (cause CRC matches, size not)
SO, CMP is perfectly correct in telling you that the size is wrong, cause it is.. !
Now surely, 131073 is a mistake, these are Binary dumps, so there is no header.
This should be 131072, no doubt...
I have not been doing SVN builds so I do not have the last & current dat..
So I can not say 'exactly' but you do..
compare j6frc10d & j6hapyhr between the 2 dats..
Has the CRC remained the same - the size changed ?
-
Maybe it is a mistake on those. Yep the size change after I done the svn update. Maybe I need to talk to them to find out why the size went up.
http://www.mameworld.info/ubbthreads/showthreaded.php?Cat=&Number=299982&page=0&view=expanded&sb=5&o=&fpart=1&vc=1 (http://www.mameworld.info/ubbthreads/showthreaded.php?Cat=&Number=299982&page=0&view=expanded&sb=5&o=&fpart=1&vc=1)
Tafoid said it been fix. It was a typo in the svn source.
-
to sum it up, case closed, cmpro is ok now.
-
Amazing:) The last question: "Parse Rom merge tags" option , enable or disable ?
-
well, actually it's your own decision :)
I did not enable it by default because the "merge" tags in MAME were too often too wrong (pointing to wrong roms, just missing or holding bad checksums) in the past. cmpro can detect such errors during datfile parsing and fixes them, but still....I did enable the chd merge tags though... :)
For romsets, I actually also like to have the correct file names which otherwise get lost due to merging...but again, it's your personal taste....
I still wonder why people always come up with the set size argument. Comeon...HD space is cheap and if you're collecting MAME sets, you should not care about a minimum space gain by using the merge tags, full merging in general or 7z...
Last but not least....people should use this forum if they have questions/problems or may found a bug...it doesn't help when endless discussions are kept on a different forum. Normally I can fix issues within a day or two...when I know about them...when I got a scenario where I can easily repeat the problem...
Thanks for any support!
-
Sure I will post here Roman when I need help or something.