EMULAB Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

The new forum is online, hope you enjoy it!

Pages: [1]   Go Down

Author Topic: Rebuilder 0.01 released  (Read 10939 times)

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Online Online
  • Posts: 3443
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Rebuilder 0.01 released
« on: 13 July 2022, 20:35 »

Rebuilder 0.01 released

Yes...something new, something fast, something I've enjoyed coding. Rebuilder is a small commandline tool which has some advantages over the rebuilder in clrmamepro. Faster, Standalone merge mode, CHD rebuild to name a few.

Read more: https://mamedev.emulab.it/clrmamepro/binaries/readme.html

Download: https://mamedev.emulab.it/clrmamepro/#downloads



Some examples:

Load a MAME 245 listxml dat, use e:\mame\roms as source, f:\sorted as destination folder, scan the source recursively and create sort the output by manufacturer and year. Rebuild in zipped split merged mode (default):
rebuilder.exe -x e:\temp\245.xml -i e:\mame\roms -o f:\sorted -r -p #MANUFACTURER#/#YEAR#

Generate standalone sets in f:\standalone, based on 245 data with roms from e:\mame\roms but limit it to machines named 194*
rebuilder.exe -x e:\temp\245.xml -i e:\mame\roms -o f:\standalone -r -m standalone -f 194*.

Trust crc32/size hash for an even faster processing...(-s none)
rebuilder -x e:\temp\245.xml -i e:\mame\roms -o f:\test1 -r -s none

Do a really slow but full sha1 input/output check and with recompressing
rebuilder -x e:\temp\245.xml -i e:\mame\roms -o f:\test2 -r -s both -c rezip

Generate a full merged output but decompressed
rebuilder -x e:\temp\245.xml -i e:\mame\roms -o f:\full -r -m full -c none

Load a softwarelist collection dat (mame.exe -listsoftware) and rebuild my unsorted stuff for all software lists at once:
rebuilder -x e:\temp\245_soft.xml -i e:\unsortedstuff -o f:\softwarelists -r

Split my sets by device, mechanical, bios folders
rebuilder -x e:\temp\245.xml -i e:\mame\roms -o f:\clean -r -p #BIOSSPLIT#

Rebuild from a to b and remove rebuilt source files
rebuilder -x e:\temp\245.xml -i e:\a -o f:\b -r -d
« Last Edit: 14 July 2022, 14:21 by Roman »
Logged


CoreyEMTP

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 19
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Re: Rebuilder 0.01 released
« Reply #1 on: 17 July 2022, 02:05 »

Interesting, looking forward to trying it as I myself have been getting back into CLI lately. Only 25 years between projects, can't wait to see what you come up with in 2050!  ;D
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Online Online
  • Posts: 3443
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Re: Rebuilder 0.01 released
« Reply #2 on: 17 July 2022, 06:47 »

Well...the 25 years in between were fille with clrmamepro and other stuff...CLI...well...for a start since I wanted to concentrate on the core. Next steps are most likely a scanner and after that a simplified UI. But not as a replacement more like an addon.
Logged

Dullaron

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 67
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.0.0 Chrome 103.0.0.0
    • View Profile
Re: Rebuilder 0.01 released
« Reply #3 on: 20 July 2022, 05:20 »

I like this better than the clrmamepro. Thank you. I'm using this.

rebuilder -x E:\MAMEDev\repack.xml -i E:\MAME_roms -o E:\MAMEDev\repack -r -s none
« Last Edit: 20 July 2022, 05:22 by Dullaron »
Logged

Dullaron

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 67
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.0.0 Chrome 103.0.0.0
    • View Profile
Re: Rebuilder 0.01 released
« Reply #4 on: 21 July 2022, 02:13 »

There is a problem. mame -listxml neogeo ng_mv4f ng_mv2f ng_mv1 ng_mv1f ng_mv1fz 2020bbh burningfpb ridheroh   > roms_bad_dump.xml
The rebuilder can't put the pieces of main roms into the clones. But the clrmamepro doesn't have that problem.

I doing something wrong?

E:\MAMEDev>rebuilder -x E:\MAMEDev\roms_bad_dump.xml -i E:\MAME_roms -o E:\MAMEDev\roms_bad_dump -r -m full -c none
[2022-07-20 21:12:57.286] [info] loading: E:\MAMEDev\roms_bad_dump.xml
[2022-07-20 21:12:57.287] [warning] 2020bbh has bad cloneof reference: 2020bb -> no parent assigned
[2022-07-20 21:12:57.287] [warning] 2020bbh has bad romof reference: 2020bb
[2022-07-20 21:12:57.287] [warning] burningfpb has bad cloneof reference: burningf -> no parent assigned
[2022-07-20 21:12:57.287] [warning] burningfpb has bad romof reference: burningf
[2022-07-20 21:12:57.288] [warning] ridheroh has bad cloneof reference: ridhero -> no parent assigned
[2022-07-20 21:12:57.288] [warning] ridheroh has bad romof reference: ridhero
[2022-07-20 21:12:57.288] [info] building view and rebuilder map: full
[2022-07-20 21:12:57.288] [info] rebuild start
[2022-07-20 21:12:57.288] [info] analyzing rebuilder source: E:\MAME_roms, please be patient
[2022-07-20 21:12:58.918] [info] analyzing rebuilder destination: E:\MAMEDev\roms_bad_dump, please be patient
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓   100%

I even tried standalone. Same issue. I don't mind keep using the clrmamepro because of this issue.

OK I think the non-merged option isn't on the rebuild. lol
« Last Edit: 21 July 2022, 02:19 by Dullaron »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Online Online
  • Posts: 3443
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Re: Rebuilder 0.01 released
« Reply #5 on: 21 July 2022, 06:10 »

"The rebuilder can't put the pieces of main roms into the clones. But the clrmamepro doesn't have that problem."
erm.....I don't get that...can you give a more detailed information here? An example? What "main roms" do you miss?

Look at your created datfile. It does not include the parent sets for e.g. 2020bb, burningf, ridheroh since you did not include it in your datfile generation
and you may loose the connection to the neogeo bios for such sets, too since the parent links to the bios via romof usually


So...please gimme some more details but I bet you find the problem inside your dat.
Logged

Dullaron

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 67
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.0.0 Chrome 103.0.0.0
    • View Profile
Re: Rebuilder 0.01 released
« Reply #6 on: 21 July 2022, 09:27 »

The main set doesn't need to be listed on the dat file. This dat file works just fine on the clrmamepro. There will be a warning on the clrmamepro. But it still send the roms from the main set over onto the clone set. Only what is missing will be copy over to clone set. Non-Merged setting on the clrmamepro.

mame -listxml neogeo ng_mv4f ng_mv2f ng_mv1 ng_mv1f ng_mv1fz 2020bbh > repack.xml

Try out this on the rebuilder and clrmamepro. You will see what I'm talking about.

The problem with the rebuilder is that it looking for main set every time. clrmamepro doesn't look for it because the main set isn't listed on the dat file.

I do know that if the neogeo set isn't listed on the dat file then the clrmamepro will send the bios over onto the main sets and clone sets.

mame -listxml 2020bbh > repack.xml

I'm not worry about it. I would recommended people to use rebuilder when not removing the main sets from the dat file. Other wise it will complains if any are removed from the dat file.

Use the clrmamepro if they just doing non-merged clone sets alone.
« Last Edit: 21 July 2022, 09:55 by Dullaron »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Online Online
  • Posts: 3443
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Re: Rebuilder 0.01 released
« Reply #7 on: 21 July 2022, 09:52 »

what is a "main set" ? Give a concrete example what you are missing in which set.
« Last Edit: 21 July 2022, 09:54 by Roman »
Logged

Dullaron

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 67
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.0.0 Chrome 103.0.0.0
    • View Profile
Re: Rebuilder 0.01 released
« Reply #8 on: 21 July 2022, 10:21 »

Not in the dat.

ROM_START( 2020bb ) Parent set/Main set.
   ROM_REGION( 0x100000, "cslot1:maincpu", ROMREGION_BE|ROMREGION_16BIT )
   ROM_LOAD16_WORD_SWAP( "030-p1.p1", 0x000000, 0x080000, CRC(d396c9cb) SHA1(47ba421d14d05b965a8d44e7475b227a208e5a07) ) /* TC534200 */
   /* also found AES set with p1 label 030-P1 on TC534200 on NEO-AEG PROG42G and m1 label 030-M1 on TC531001 on NEO-AEG CHA42G; other chip labels are the same */

   NEO_SFIX_128K( "030-s1.s1", CRC(7015b8fc) SHA1(8c09bc3e6c62e0f7c9557c1e10c901be325bae7f) ) /* TC531000 */

   NEO_BIOS_AUDIO_128K( "030-m1.m1", CRC(4cf466ec) SHA1(6a003b53c7a4af9d7529e2c10f27ffc4e58dcda5) ) /* TC54H1000 */

   ROM_REGION( 0x200000, "cslot1:ymsnd:adpcma", 0 )
   ROM_LOAD( "030-v1.v1", 0x000000, 0x100000, CRC(d4ca364e) SHA1(b0573744b0ea2ef1e2167a225f0d254883f5af04) ) /* TC538200 */
   ROM_LOAD( "030-v2.v2", 0x100000, 0x100000, CRC(54994455) SHA1(76eb62b86e8ed51a77f44313d5cc8091b3f58d57) ) /* TC538200 */

   ROM_REGION( 0x400000, "cslot1:sprites", 0 )
   ROM_LOAD16_BYTE( "030-c1.c1", 0x000000, 0x100000, CRC(4f5e19bd) SHA1(ef7975c4b33a7aea4a25a385f604799f054d3200) ) /* Plane 0,1 */ /* TC538200 */
   ROM_LOAD16_BYTE( "030-c2.c2", 0x000001, 0x100000, CRC(d6314bf0) SHA1(0920cc580d7997fcb0170dd619af2f305d635577) ) /* Plane 2,3 */ /* TC538200 */
   ROM_LOAD16_BYTE( "030-c3.c3", 0x200000, 0x100000, CRC(47fddfee) SHA1(297c505a63448c999a2510c27bf4549102134db8) ) /* Plane 0,1 */ /* TC538200 */
   ROM_LOAD16_BYTE( "030-c4.c4", 0x200001, 0x100000, CRC(780d1c4e) SHA1(2e2cf9de828e3b48642dd2203637103438c62142) ) /* Plane 2,3 */ /* TC538200 */
ROM_END

In the dat.

ROM_START( 2020bbh ) Clone set.
   ROM_REGION( 0x100000, "cslot1:maincpu", ROMREGION_BE|ROMREGION_16BIT )
   ROM_LOAD16_WORD_SWAP( "030-pg1.p1", 0x000000, 0x080000, BAD_DUMP CRC(12d048d7) SHA1(ee0d03a565b11ca3bee2d24f62ff46a85ef18d90) )
   /* Chip label p1h does not exist, renamed temporarily to pg1, marked BAD_DUMP. This needs to be verified. */

   NEO_SFIX_128K( "030-s1.s1", CRC(7015b8fc) SHA1(8c09bc3e6c62e0f7c9557c1e10c901be325bae7f) ) /* TC531000 */

   NEO_BIOS_AUDIO_128K( "030-m1.m1", CRC(4cf466ec) SHA1(6a003b53c7a4af9d7529e2c10f27ffc4e58dcda5) ) /* TC54H1000 */

   ROM_REGION( 0x200000, "cslot1:ymsnd:adpcma", 0 )
   ROM_LOAD( "030-v1.v1", 0x000000, 0x100000, CRC(d4ca364e) SHA1(b0573744b0ea2ef1e2167a225f0d254883f5af04) ) /* TC538200 */
   ROM_LOAD( "030-v2.v2", 0x100000, 0x100000, CRC(54994455) SHA1(76eb62b86e8ed51a77f44313d5cc8091b3f58d57) ) /* TC538200 */

   ROM_REGION( 0x400000, "cslot1:sprites", 0 )
   ROM_LOAD16_BYTE( "030-c1.c1", 0x000000, 0x100000, CRC(4f5e19bd) SHA1(ef7975c4b33a7aea4a25a385f604799f054d3200) ) /* Plane 0,1 */ /* TC538200 */
   ROM_LOAD16_BYTE( "030-c2.c2", 0x000001, 0x100000, CRC(d6314bf0) SHA1(0920cc580d7997fcb0170dd619af2f305d635577) ) /* Plane 2,3 */ /* TC538200 */
   ROM_LOAD16_BYTE( "030-c3.c3", 0x200000, 0x100000, CRC(47fddfee) SHA1(297c505a63448c999a2510c27bf4549102134db8) ) /* Plane 0,1 */ /* TC538200 */
   ROM_LOAD16_BYTE( "030-c4.c4", 0x200001, 0x100000, CRC(780d1c4e) SHA1(2e2cf9de828e3b48642dd2203637103438c62142) ) /* Plane 2,3 */ /* TC538200 */
ROM_END
« Last Edit: 21 July 2022, 10:27 by Dullaron »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Online Online
  • Posts: 3443
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Re: Rebuilder 0.01 released
« Reply #9 on: 21 July 2022, 10:43 »

Ah.... wait....Now I think I see what you mean.....you miss the parent roms...well..of course you do because your dat does not include the parent.

So...ok....I think if a "ridheroh has bad cloneof reference: ridhero -> no parent assigned" message appears, the single roms inside it should get their merge attribute removed since there is no more relationship....I will see what I can do....
« Last Edit: 21 July 2022, 11:46 by Roman »
Logged

Dullaron

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 67
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.0.0 Chrome 103.0.0.0
    • View Profile
Re: Rebuilder 0.01 released
« Reply #10 on: 22 July 2022, 04:30 »

no worry
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Online Online
  • Posts: 3443
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Re: Rebuilder 0.01 released
« Reply #11 on: 22 July 2022, 13:43 »

Well ok while it is easy to "fix" the bad "cloneof" attribute information from 2020bbh by clearing it and clearing the obsolete "merge" attribute information from its roms (since it's not a clone anymore), it however brings up some issue:

What about the merge information which is connected to bios roms. Since you've included neogeo in your dat, they should somehow stay. Removing the merge for them, too, would include them in your rebuilt 2020bbh set.

However since 2020bbh's "romof" information is also bad you can't track the bios machine (since it is lost by removing the parent which had the romof="neogeo").
Then you would say: ok...only remove the merge information from the roms which don't have a "bios" attribute...fine...most will be ok then but that would still include sfix.sfix, 000-lo.lo and sm1.sm1 in the set since they for whatever reason don't have a bios attribute.

Need to find a sutiable way for this...don't really like the way to check the single roms against possible available bios roms....but most likely that would be the only way to fix a bad romof attribute and reconstruct a bios assignment.


Anyhow....this is only for dats which do have bad structures.....pretty special to start with :-) ...will look at it again after summer holidays.
Logged

Dullaron

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 67
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.0.0 Chrome 103.0.0.0
    • View Profile
Re: Rebuilder 0.01 released
« Reply #12 on: 22 July 2022, 14:14 »

Here is a copy from the dat file. In a txt file. The way MAME made this.

Both have biosset info.

You have a good point on the bios. lol
« Last Edit: 22 July 2022, 14:25 by Dullaron »
Logged

Roman

  • Global Moderator
  • Member
  • ***
  • Karma: 120
  • Online Online
  • Posts: 3443
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.5060.114 Chrome 103.0.5060.114
    • View Profile
Re: Rebuilder 0.01 released
« Reply #13 on: 22 July 2022, 15:08 »

A bios assignment is done by tracking romof attributes.
So normally you got 2020bbh romof 2020bb and 2020bb romof neogeo. So you know that 2020bbh is connected to the neogeo bios set.
Now you removed 2020bb and you keep only 2020bbh....and if you look at the following roms in that set, how do you decide if this belongs to the removed parent 2020bb or to the missing link to the bios?

<rom name="sfix.sfix" merge="sfix.sfix" size="131072" crc="c2ea0cfd" sha1="fd4a618cdcdbf849374f0a50dd8efe9dbab706c3" region="fixedbios" offset="0"/>
<rom name="000-lo.lo" merge="000-lo.lo" size="131072" crc="5a86cff2" sha1="5992277debadeb64d1c1c64b0a92d9293eaf7e4a" region="spritegen:zoomy" offset="0"/>
<rom name="sm1.sm1" merge="sm1.sm1" size="131072" crc="94416d67" sha1="42f9d7ddd6c0931fd64226a60dc73602b2819dcf" region="audiobios" offset="0"/>


Well, you'd say: I can simply lookup the sha1 in all roms and see that it belongs to the neogeo set......but I'd like to prevent such "guessing"



So the story is: If you have a bad cloneOf assignement: what to do? In Rebuilder 0.01 your problem is that the parser warns you and the definition stays untouched, i.e. all "merge" attributes stay as they are you so in the end you don't get your 2020bb files.
"Fixing" this bad cloneOf would mean to clear it and clean the merge attributes...cleaning them all would also clean the ones belonging to the bios....so you would end up with a set which got all bios roms included....you don't want that since you got neogeo bios in the dat listed....cleaning the merge attributes for the ones which don't have a bios attribute would work fine...but the upper 3 don't have one...and so they are included....


or I don't care at all since it's a corruption in the dat, so garbage in -> garbage out.....
« Last Edit: 22 July 2022, 16:31 by Roman »
Logged

Dullaron

  • Member
  • *
  • Karma: 0
  • Offline Offline
  • Posts: 67
  • Operating System:
  • Windows NT 10.0 Windows NT 10.0
  • Browser:
  • Chrome 103.0.0.0 Chrome 103.0.0.0
    • View Profile
Re: Rebuilder 0.01 released
« Reply #14 on: 22 July 2022, 22:45 »

You are right. Removing parent will break the scanning reading from the dat when using the rebuilder.

I will keep testing. 2020bb and 2020bba. Keep the parent in the dat this time.
Logged
Pages: [1]   Go Up
 

Page created in 0.11 seconds with 21 queries.