EMULAB Forum

clrmamepro [English] => clrmame Discussion => Topic started by: newuzer on 30 July 2020, 06:49

Title: CMP 4.037a scanner ROMs deleted
Post by: newuzer on 30 July 2020, 06:49
Hi Roman,

After rebuilding HBMAME 0.222 to 0.223, 2 mslug3 ROMs that were part of 0.222 were mysteriously missing after scanning/fixing.
I can reproduce the issue.
 
The corresponding datfile confirms that both ROMs are part of the "HBMAME 0.222 ROMs(merged)" set as:

Code: [Select]
<rom name="mslug3h58\256h58.p2" size="4194304" crc="15bb1f0d" sha1="bf0a6bf38a4addcf9a7a08b04943594666347d6a" />
<rom name="mslug3h62\256h62.p1" size="1048576" crc="61f2a873" sha1="9e9804ffc94fe3262bb4e8efe1ba7e313d4ab37e" />

I've copied only mslug3.zip from the 0.222 set to a new folder "HBMAME 0.223 ROMs (merged) - mslug3" and created a "HBMAME 0.223 ROMS (merged) - mslug3.xml" datfile (attached).

If you don't have it, you can also rebuild it with the attached "HBMAME 0.222 ROMs (merged) - mslug3.xml" datfile.
Load the datfile in CMP 4.037a.
Point at the new folder mentioned above.
Run a scan with all fix options checked.
When prompted to fix: Yes To All
Rebuild from backup (none rebuilt) and the update (3 added).
Ended up with the 2 missing ROMs:

Code: [Select]
Metal Slug 3 (NGM-2560) [folder: mslug3 - size: 660mb]
missing rom: mslug3h58\256h58.p2 [size: 4194304] [CRC32: 15bb1f0d] [SHA1: bf0a6bf38a4addcf9a7a08b04943594666347d6a]
missing rom: mslug3h62\256h62.p1 [size: 1048576] [CRC32: 61f2a873] [SHA1: 9e9804ffc94fe3262bb4e8efe1ba7e313d4ab37e]

CMP dialog output:
Code: [Select]
Set: Metal Slug 3 (NGM-2560)
Name: mslug3
File: HBMAME-ROMs\HBMAME 0.223 ROMs (merged) - mslug3\mslug3.zip

Do you want to rename the following wrong named ROMs?

mslug3h01\256h01.p1 -> mslug3ern08\256h01.p1
mslug3h06\256h06.p1 -> mslug3ern05\256h06.p1
mslug3h09\256h09.p1 -> mslug3ern09\256h09.p1
mslug3h40\256sold.c1 -> mslug3esrp02\256sold.c1
mslug3h40\256sold.c2 -> mslug3esrp02\256sold.c2
mslug3h40\256sold.c3 -> mslug3esrp02\256sold.c3
mslug3h40\256sold.c4 -> mslug3esrp02\256sold.c4
mslug3h40\256sold.m1 -> mslug3esrp02\256sold.m1
mslug3h40\256sold.s1 -> mslug3ern02\256sold.s1
mslug3h41\256h41.p1 -> mslug3ern04\256h41.p1
mslug3h42\256h42.p1 -> mslug3ern03\256h42.p1
mslug3h46\256h46.m1 -> mslug3ern02\256h46.m1
mslug3h46\256h46.p1 -> mslug3ern02\256h46.p1
mslug3h46\256h46.v4 -> mslug3ern02\256h46.v4
mslug3h48\256h48.p2 -> mslug3esrp05\256h50.p2
mslug3h51\256h54.c7 -> mslug3ern01\256h54.c7
mslug3h51\256h54.c8 -> mslug3ern01\256h54.c8
mslug3h51\256nd.p1 -> mslug3h48\256nd.p1
mslug3h52\256h52.p1 -> mslug3ern01\256h52.p1
mslug3h56\256h56.p2 -> mslug3ern01\256h56.p2
mslug3h57\256h59.p2 -> mslug3esrp01\256esrp01.p2
mslug3h58\256h58.p2 -> mslug3h48\256h48.p2
mslug3h60\256h60.p2 -> mslug3ern03\256ern03.p2
mslug3h61\256h61.p2 -> mslug3esrp02\256h61.p2
mslug3h62\256h62.p1 -> mslug3esrp04\256esp04.p1
mslug3h62\256h62.p2 -> mslug3esrp04\256esp04.p2
mslug3h63\256h63.p2 -> mslug3ern04\256h63.p2
mslug3h64\256h64.p1 -> mslug3h56\256h56.p1
mslug3h65\256h69.p2 -> mslug3ern02\256h60.p2
mslug3h66\256h66.p1 -> mslug3h57\256h57.p1
mslug3h66\256h66.p2 -> mslug3h57\256h57.p2
mslug3h67\256h67.p1 -> mslug3h58\256h58.p1
mslug3h67\256h67.p2 -> mslug3h58\256h58.p2
mslug3h68\256h68.p2 -> mslug3h59\256h59.p2
mslug3h68\neo-sma_h68 -> mslug3h59\neo-sma_h68
mslug3h70\256h70.p1 -> mslug3ern06\256h61.p1
mslug3h73\256h73.p1 -> mslug3h62\256h62.p1
mslug3h74\256h74.p1 -> mslug3h63\256h63.p1

---

Set: Metal Slug 3 (NGM-2560)
Name: mslug3
File: HBMAME-ROMs\HBMAME 0.223 ROMs (merged) - mslug3\mslug3\mslug3h48\256h48.p2

Do you want to add the missing ROM?

---

Metal Slug 3 (NGM-2560) [folder: mslug3 - size: 660mb]
missing rom: mslug3ern07\256ern07.p1 [size: 1048576] [CRC32: ccfafa60] [SHA1: 68f0d188b89ccd0d3c9ae1c9f2508321acbd8871]
missing rom: mslug3ern10\256ern10.p1 [size: 1048576] [CRC32: 30a6aa55] [SHA1: 3000d68e6e22ec8b588c7099fac3fb3ee74fbb5d]
missing rom: mslug3ern11\256ern11.p1 [size: 1048576] [CRC32: 79c73bb5] [SHA1: f50e69e308c89badd2e12518d0c5437e50a185d2]
missing rom: mslug3h58\256h58.p2 [size: 4194304] [CRC32: 15bb1f0d] [SHA1: bf0a6bf38a4addcf9a7a08b04943594666347d6a]
missing rom: mslug3h62\256h62.p1 [size: 1048576] [CRC32: 61f2a873] [SHA1: 9e9804ffc94fe3262bb4e8efe1ba7e313d4ab37e]

---

Metal Slug 3 (NGM-2560) [folder: mslug3 - size: 660mb]
missing rom: mslug3h58\256h58.p2 [size: 4194304] [CRC32: 15bb1f0d] [SHA1: bf0a6bf38a4addcf9a7a08b04943594666347d6a]
missing rom: mslug3h62\256h62.p1 [size: 1048576] [CRC32: 61f2a873] [SHA1: 9e9804ffc94fe3262bb4e8efe1ba7e313d4ab37e]

I think it's the renaming of these 2 ROMs where things go wrong:
mslug3h58\256h58.p2 -> mslug3h48\256h48.p2
mslug3h62\256h62.p1 -> mslug3esrp04\256esp04.p1
I'm not sure why CMP comes to the conclusion that they need to be renamed.

NU.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 30 July 2020, 15:46
>I'm not sure why CMP comes to the conclusion that they need to be renamed.

Because they changed their name (you need to go by sha1)

.222
<rom name="mslug3h58\256h58.p2" size="4194304" crc="9e2064e6" sha1="d807eb56aebd7f5e8b43b67291856ebb07130c1b"/>
<rom name="mslug3h48\256h48.p2" size="4194304" crc="7593474c" sha1="fcdd76013069eff64dc6842c672870854a53c0f2"/>

.223
<rom name="mslug3h58\256h58.p2" size="4194304" crc="15bb1f0d" sha1="bf0a6bf38a4addcf9a7a08b04943594666347d6a" />
<rom name="mslug3h48\256h48.p2" size="4194304" crc="9e2064e6" sha1="d807eb56aebd7f5e8b43b67291856ebb07130c1b" />


Since I don't have the romsets (neither .222 or .223) I can't check deeper.....but I don't think something is lost. There are cases where things are put to backup and need a 2nd scan to get readded or get rebuilt......You said you tried that and didn't work....so I first need the files for doing another check.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 30 July 2020, 21:16
ok it seems to be a problem within the used zip library's rename operation when having constellations where you got sort of circular renames
...need some more time to dig deeper into this. By the way, only zip is affected, rar/7z archives work fine in this scenario.

Thanks for reporting.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 31 July 2020, 11:06
Fixed for the next version..thanks again
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: newuzer on 07 August 2020, 06:33
Hi roman,

Just to let you know before I'm able to check if I can reproduce.
There appears to be an issue with 4.038 after fixing this problem.
Several people reported (the same) that after scan/fixing MAME 0.222 ROMs (merged/split) to 0.223 with 4.038 these sets turned out wrong:
Quote
TorrentCheck reported:
Wrong file size: "D:\MAME\ROMs\deathrac.zip" wrong: 2061 right: 2295
Wrong file size: "D:\MAME\ROMs\v4mdice.zip" wrong: 4580161 right: 4167372
Wrong file size: "D:\MAME\ROMs\v4monte.zip" wrong: 2341626 right: 2332683

Member quote:
Yeah the funny thing, I looked through my CMP backup folder for 0.223 roms and deathrac.zip was in there.  I opened the zip and both the roms in question it was complaining about were there.  I extracted them and pointed CMP to them but it didn't even care they were there.

The sets compared to were created with 4.037a.
Doing a check now...

[edit]
I can reproduce there is an issue, not with deathrac but with the two v4m sets.
The resulting zipfiles v4mdice.zip and v4monte.zip created with both 4.037a and 4.038 have a different size (after torrentzipping).
However if I cross-check both zipfiles scan fine with 4.037a and 4.038.
[/edit]

NU.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 07 August 2020, 06:44
TorrentCheck? What does a torrent check size compare tell you about the correctness of romsets? Nothing. Especially not the size of the zip itself.
A circular rename renames in 2 steps, so it may even alter the order in the zip twice.

So, please give me a repeatable example with standard clrmamepro and such sets and tell me where it goes wrong.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: newuzer on 07 August 2020, 08:05
I have created a folder for you in my ftp/upload directory with the files and information to reproduce.

NU.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 07 August 2020, 08:20
Thx, I will have a look...but again...torrentzip is not something I care for....and especially not the size of a zipfile.

It's only important that the zips are correct in terms of content.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: newuzer on 07 August 2020, 08:23
Torrentzip is pretty vital for our use.
The "other" ROM managers create the 4.037a result.
I think it's very important to have a consistent result across versions and even ROM managers.

NU.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 07 August 2020, 09:02
But if the zip file content is the same as before (no additional files, no missing files, same files bytewise), why does torrentzip create a different output then...sounds to me more like a torrentzip weirdness....or do you see differences in the content?
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 07 August 2020, 11:33
as discussed...seems they are caused by some dupe entries in the zip.
Will do addtional testing and try to fix it as quick as possible.

By the way, "Settings->Compressor->General->list archived files with double names in warnings window" helps you to find such dupes.

Seems that only v4mdice and v4monte got the dupes problem, in the meantime I fixed the problem and will release a new version soon.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: newuzer on 07 August 2020, 14:34
Test version fixed the problem.

NU.
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: Roman on 07 August 2020, 15:02
thanks for reporting/testing...now get the update 8)
Title: Re: CMP 4.037a scanner ROMs deleted
Post by: newuzer on 07 August 2020, 16:00
Grabbed!
Thanks for the quick fix, Roman!

NU.