Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

2661

clrmame Discussion / Re: Work In Progress

« on: 11 August 2011, 20:49 »

well, I quickly wrote a zip parser based on the already used ziparchive lib and it seems to solve the remaining issues with extra-field utf8 etc....(I need some confirmation from oxyandy though)

Problem is....I need to get rid of all the other internal in-place-rename and no-recompress copy routines...The good news is that the library already got such functions, however I need to adopt them...which takes time...which I currently don't really have...

I will have a week end of August....and some hours here and there till then....

2662

clrmame Discussion / Re: Work In Progress

« on: 11 August 2011, 19:25 »

ok....I won't write apos anymore for an apostrophe...

now back to the general problem: it seems that tools like Winzip do use the extra fields for utf8 filenames. Funnily enough I can use Winzip without doing that and ziparchive lib I'm using can use this mode fine...
So cmpro created files (forget about no-recompress at the moment) work fine in Winzip and within cmpro.
Problematic are files which exist before you run cmpro on them...

Either:
- they got a local code page encoding, then cmpro reads them in, converts them as utf8 (which is wrong) and you got wrong characters. Rezipping them with cmpro will fix this.
- or they use the extra field to store utf8 additional information...which my current parser does not handle....I try to update it...or switch to the ziparchive readers completely...

Thanks for testing...

2663

clrmame Discussion / Re: Work In Progress

« on: 11 August 2011, 10:15 »

Again...non-utf8 encoding is not an option here. It will then simply use local page encoding which may work on your side but nowhere else. The goal is to use utf8 stored names and give a s**t on local page encoding.

Zip writing is not a problem. It uses the ziparchive lib for saving (unless you use 'no recompress' in the rebuilder, then the name is converted internally and not in the lib!!!) and that stores the filename utf8 encoded and winzip, winrar and 7z shows the name correctly. It uses the method to store this without EXTRA field usage which all 3 major programs handle fine.

2664

clrmame Discussion / Re: Work In Progress

« on: 11 August 2011, 09:56 »

thanks! good to know...so it seems it can't convert the chars...so the issue is not in dir2dat but in the reader

I will check if you can specify a default char (like a ?) in case of a not converted char...I assume you've mailed me that "\哈哈哈 (ZIP MADE BY WinZip 15.5).zip"....please do so...I bet it uses the extrafield for additional information

2665

clrmame Discussion / Re: Work In Progress

« on: 11 August 2011, 09:43 »

yeah yeah..that's the option I was talking about...good to see it's checked...

Now please drag'n drop the file from your dir2dat source (the one which creates the empty filename) into the about window...
a window should open, listing all entries of the zip...I wonder if the name is listed there (correctly).
If it is listed there, then dir2dat somehow cuts off the name....if it's not listed there, then it's related to the encoding / zip reader.

2666

clrmame Discussion / Re: Work In Progress

« on: 11 August 2011, 09:29 »

Check winzips15 options if you got "store filenames as utf8" checkbox ticked....

And for another test: drop the zipfile which gives you an empty filename in dir2dat into the about window...does it list an empty filename there, too?

2667

clrmame Discussion / Re: Work In Progress

« on: 11 August 2011, 07:50 »

well well well...Looks hard times will come up

while testing I commonly see that users react on wrong character encoding when working with zipfiles which were created outside of cmpro...

So...let's start by saying that utf8 and zip files are two worlds...there is no real standard how files are stored as utf8 encoded.

Winzip15 allows storing the filename as utf8 with no extra information. Simply store the utf8 hex bytes and you're done...fine...cmpro handles that..

If you use an packer version or a different product the name is most likely stored with the local code page encoding....so what will happen, cmpro reads the zipfile in and converts the filename based on the assumption that it's utf8 encoded...which ends with an output you did not expect. If this happens, well, tough luck, let cmpro recreate the file for you....

I'm currently checking if Winrar zipfiles have similar issues or if they are using the 2nd way to store utf8 filenames in zips (by the way, officially winrar supports this for zips since version 3.80) which is by using the zip extra field for some encoding information. If so, I have to update my zipreader a bit to use the extrafield information...

There were comments that the encoding goes nuts after torrentzipping the file..well..then the torrentzip guys should double check their way of encoding names (I assume they do local code page encoding) as well...but I will check if they maybe use the extra-field method already.

rar and 7z archives seem to have no issues at all anymore since the last build....and Winzip created zips and cmpro created zips either.

...so the next step will be to double check winrar zips...

Keep in mind that the goal is to support utf8 encoding, not to support any local page encoding...

2668

clrmame Discussion / Re: Work In Progress

« on: 10 August 2011, 19:36 »

ok...some news...somehow related to utf8

- made stats.ini an utf8 file with BOM (to show middle point at the beginning correctly)
- fixed some name conversion for 7z files
- fixed a typo which caused the batcher to not correctly setup rompaths
- xml files are now loaded as utf8, no matter what encoding you specify

Generally you might face issues when you got existing archives which have the name stored in a local-page encoding...and then they get read as utf8 within cmpro...so you run into some wrong conversions...but I hope over time people get rid of such issues...

2669

clrmame Discussion / Re: Work In Progress

« on: 09 August 2011, 20:19 »

Well, actually I already tried that in the past and complaint that the file either needs a BOM or the encoding attribute...but fine...if the XML standard says that a non existing encoding attribute falls back to utf8, I can easily add that...

2670

clrmame Discussion / Re: Work In Progress

« on: 04 August 2011, 19:59 »

For those who are testing:

Keep in mind that dats (if they use non-ascii chars) need to be saved as utf8 with or without a BOM (ByteOrderMark). If you don't use a BOM, be sure that your xml datfile holds an encoding="utf-8" attribute within the xml tag at the beginning.
XML dats are prefered of course, however old style dats should work as well (when saved with BOM).
Again, I can recommend notepad++ for easy saving and editing dats...

...and don't expect feedback before next week

2671

clrmame Discussion / Re: Profiler slow to load

« on: 04 August 2011, 19:23 »

Turn off virusscanners completly (for a test) and be sure you didn't use 3rd party tools to mass-create cmpro profiles...
Last time I've checked it it loaded something >3000 profiles within seconds...
all I can say for now...

2672

clrmame Discussion / Re: Work In Progress

« on: 03 August 2011, 17:29 »

well, nothing really new regarding utf8....only made a clean compile with a clean new solution setup in my Visual Studio....actually I somehow screwed up my old one and it had major problems with precompiled headers....hehehe
so..new setup...working fine...

so...I guess next week could be a good start for some testing...if you're interested, let me know your email address...can't guarantee that everyone gets a testversion .... and as I said...earliest is somewhen next week...

By the way, if you got some utf8 dats, please send them in....best would be if the crc32/sha1/sizes match MAME roms ....and the names / description / manufacturer tags etc  could be something chinese/japanese/etc....



On a sidenote .... if you don't want to see a winrar window popping up when adding/deleting files, add a -ibck in the rar commandline params...

2673

clrmame Discussion / Re: Enhancement request ;-)

« on: 30 July 2011, 05:55 »

well, actually I prefer that people take the profler.xml file, run it through some xslt transformations and create their own reports anytime, anyplace and create a html output they prefer...


but I put your request on my list....but currently utf8 is prio #1

2674

clrmame Discussion / Re: Work In Progress

« on: 28 July 2011, 20:01 »

ok...back to the main topic...utf8...

one thing I wondered about was....what happens if MAME's -listxml output prints out asian characters....and cmpro's profile points to the mame binary (ok, internally it calls MAME and redirects its output..).

...So I tested it....

1st step...picked a set (for easyness I simply used the first in the xml "005" from segag80r.c) and changed a rom name to something in chinese...saved it as utf8 without BOM (with notepad++), recompiled MAME....a -listxml output lists the asian characters..looks good...

2nd step...since cmpro will parse xml datfiles as utf8 only when they got a BOM or when the encoding is specified in the XML tag, I changed info.c to add an encoding="utf-8" attribute, recompiled MAME, -listxml shows it...fine

3rd step...let cmpro import the data directly.......the stdout redirector did a good job....it works

So...actually if anyone ever decides to update MAME to list set names, rom names, descriptions etc in the original language...fine....DO SO!

http://mamedev.emulab.it/clrmamepro/mame_utf8.png

2675

clrmame Discussion / Re: Work In Progress

« on: 28 July 2011, 11:56 »

Enable Profiler -> Options -> parse Rom Merge Tags if you want to get rid of the dupes.
Actually I think you can force to split-merged sets if you want to avoid the removal of parent/clone relationships in case of identical names for non-identical files...but I have to check that....

2676

clrmame Discussion / Re: Work In Progress

« on: 27 July 2011, 21:36 »

so, the next entry in the diary...well...nothing really special today...

I've tested some standard zip/rar/7z if they work fine (ehehe...not that I get multibytes for ascii7 chars now....) and checked under which circumstances Winzip creates utf8 encoded files with extra-field usage...actually I wasn't able to produce one...so again...I will stick to the non-extra-field-usage method to use utf8.

I more used today's little time to align scanner's 'allow not separated bios sets' and rebuilders 'split bios sets' option...they are both named identical now (split bios sets). Also I started to remove the scanner advanced '* SysDefPath' options from the UI......they will be internally enabled if sysdefpaths are setup...which makes more sense in my opinion....checking other options as well..maybe some become obsolete to be set by the user....time will tell...

2677

clrmame Discussion / Re: Work In Progress

« on: 27 July 2011, 15:59 »

then disable rebuilder's recompress files option

2678

clrmame Discussion / Re: clrmamepro hardware question

« on: 27 July 2011, 13:17 »

Well, generally more RAM is a good thing for your PC...if you got less than 4GB..above...hm..nah..won't help you that much..

Generally, it highly depends on how sets are organized in the dat.

For example, if you got a 1-set-dat like "Flyers", all files are in one set, i.e. you do renames/fixes on one big zip...that takes time since it needs to reorganize the zip over and over again. An option could be to switch to decompressed sets in that case, but that would require crc32 calculation on files...which takes time again...

On the other hand if you got a 1-file-1-set dat where a set only consists of 1 file...there a rename is very fast since it doesn't have to care about other files.

Scanning/Rebuilding/Fixing options which are a speed drop are:
- rebuilder's "recompress files" option
- any sha1/md5 or deep crc32 additional checks (rebuilder advanced, scanner/checksum)
- header usage (it then has to decompress files, skip header, calc crc32)


So...before you buy new hardware, better check the mentioned options (turn them off if possible ) and check if your zips which take long are such 1-zip-10000 files ones...they simply take time....(until I find time to optimize their handling).

2679

clrmame Discussion / Re: Work In Progress

« on: 26 July 2011, 20:50 »

ok...some progress...(does somebody actually reading this diary??)

So....yesterday I did hook up the latest zipclass lib but today I had several issues with utf8 names in created zips...after some research I found out that I somehow added a wrong version (doooh). A clean remove and reinstall of the latest lib solved all issues (Thanks again Tadeusz!).

All issues solved? well...actually I found out another thing...there are several ways for 'standard' zipfiles to handle utf8 encoded names....one method (which I prefer) is simply store names in utf8 and hope that applications handle it correctly. Winzip (15.x), 7z (9.x) and Winrar (4.x) do...so I stick to this method...The other method is to store the filename differently and keep information about encoding in the zip structure extra fields....actually this also makes the zipfile larger...

Now that it seems that we have full utf8 support for all 3 archive types, I also cleaned up the settings->compressor screen:

* oem2ansi conversion is gone (yes...live with it...use utf8!)
* zip compression level is gone (internally 'best' is used which corresponds to '9')...I don't see any reason why this should be selectable by the user. You can now start a discussion about torrentzip or why not using bzip2 as compression method (which newer zip programs can handle)....but that's something for the bin or the future...
* zip flush option...removed...It's a relict from years ago where people had slow hds and faulty chipsets....
* zip buffer option...removed...In our days I don't think increasing this will give you a speed boost....I may try some internal testing on different values...


So....that's it for today...Next steps will be testing, testing, testing...again, I got no deadline for a release in my mind yet...

Some other remarks: Well, I'm working with Windows7 (ultimate) and it does not have any problems showing asian characters...I know form XP that you need to first install some asian-related charsets/libs so you can view such characters correctly. Windows does that for you...in your system regional settings you find a checkbox to enable asian-character-support.
Datauthors may wonder how to work with utf8 dats...well...get a good texteditor. While I'm a big fan of Textpad, I have to say that for such tasks, Notepad++ (Yes, plus plus, not the notepad.exe from your standard Windows ) is great to use since it offers you utf-8 saving/loading options...


Ok...that's it for now....I only wonder what happens if MAME.exe's -listxml prints out utf8 names on stdout and I redirect it and read it in....yay...something to test

2680

clrmame Discussion / Re: Work In Progress

« on: 25 July 2011, 21:13 »

no big news today...did some testing of the internal zip routines, some fixing here and there, removed the usage of any oem2ansi conversion, zip flush, zip buffer settings and added the latest ziparchive lib....compiles and links fine. However it seems that I need to pass some special parameters so it works fine with utf8....ok...reading docs now....