Topic: Open source on Github (Read 13904 times)

SpaceAgeHero · « **on:** 21 February 2016, 06:18 »

Hi Roman,
wouldn't it be awesome to have clrmamepro running on Linux natively? Isn't it time to move away from Windows?

How about opening source on Github? It'll be fun.

Roman · « **Reply #1 on:** 21 February 2016, 18:23 »

Time to move away from Windows...damn...and I thought Desktop Linux failed....

Enough kidding...no, the current core won't run on Linux kernels natively nor will it be easy to update the code so that it runs there.....for now it remains closed source.

gatti · « **Reply #2 on:** 18 March 2018, 07:19 »

Hello,

Moving to .NET could be a first step to the universal app, Linux and MacOS X users will be able to run cmpro using Mono instead of Wine which is painful as soon as an app is using lot of disks I/O

btw, as I talk about I/O : cmpro seems to be incredibly slow on empty subdirs when "looking for unknown sets", that strange behavior is far more visible thru samba shares, at a point that seconds become minutes

regards

Roman · « **Reply #3 on:** 18 March 2018, 08:55 »

you mean an empty folder? with no files or a folder with no files belonging to the currently loaded database?
wine or native windows?

gatti · « **Reply #4 on:** 18 March 2018, 10:50 »

I mean all setnamed folders that should contain chd files inside rom path, except that almost chd were removed, and so those folders are currently empty
This happens on clrmamepro thru wine with direct access on a mounted ntfs drive *and* also on windows native when accessing thru samba share to that same mounted ntfs drive (using ntfs-3g fuse filesystem), the cpu goes above 80% on mount.ntfs-3g process during that phase (and it's significantly faster on checking unknown zip files than those empty directories)
Please note that I'm currently converting that drive from ntfs to ext4
I tried to scan the same rom dir with RomVault (using Mono and also native windows thru samba share), and got no general problem of performance, but I don't know what is exactly done while scanning with that rom manager...
Please also note that the rom path on NTFS drive is a SYMLINKD on another ROM path on that same drive

Roman · « **Reply #5 on:** 18 March 2018, 12:08 »

so you got for example "your_rompath/area51/" (for the chd) but nothing ( no files at all) in that subpath and area51.zip with the roms elsewhere...
I will have a look.. ..

gatti · « **Reply #6 on:** 18 March 2018, 18:04 »

Yes, that’s right
It’s faster with ext4, but still present, it should be also visible on windows with slow drives (let’s say on usb2 keys for example)

Roman · « **Reply #7 on:** 18 March 2018, 19:41 »

hmm...I've created a new rompath and batch created something around 700 empty folders in there for the chd sets in mame and started a scan with that one as rompath...and it flies right through it...no delay, no lag....
that's with native windows 10

gatti · « **Reply #8 on:** 19 March 2018, 08:17 »

That may not be noticeable if your hardware is too fast, I started to see a problem when I was working on slow hardware, but even under wine with ext4 on sata drive or usb3 drive, that won't be visible, but if you use usb2 port, and/or an old drive, and/or a shared network, or whatever which slow down I/O you should start to notice that "looking for unknown sets" phase will be slow on empty directories, and will accelerate on zip files...
I tried ntfs and ext4 (but not fat), wine and native, local and shared network... the problem is present everywhere at different degrees when using slow hardware

Another odd problem on slow hardware (in my case it's an Intel Celeron J800 with 2GB DDR3@1333) especially with wine, when I click on "Scan" or "New Scan" with a profile that use a full mame software list, I have to wait lot of minutes (around 30mn) before the progress bar start to show with "Creating HashTables"... During this time the only thing I see is cmpro's memory usage growing slowly from 270MB up to ~1GB , no disk io, and its process at 100%, I suppose you are making many small struct allocation in memory for each rom/set... is it possible to look at that part to see if it's possible to optimize? Please note that this 30mn wait does happen with a freshly started cmpro, it does not happen if I start a scan again, or if I reload the same profile

Thank you

Roman · « **Reply #9 on:** 19 March 2018, 09:42 »

Well, yes, I'm on a rather fast PC (I7-8700K, 16GB DDR4@2666) but still, an empty folder shouldn't have any impact at all, since well, cmpro will check the files in there during the unneeded check....no files, no check...

Sure, during the Creating HashTables process a hashtable containing any rom plus hash is generated, but still, since the final size is known it preallocates it first and then fills it up.

When I find a little time I can try to optimize it....

gatti · « **Reply #10 on:** 19 March 2018, 10:21 »

So if I understand well the situation, on my J1800 with wine (debian buster), the preallocation takes 30mn, then the progress bar show up, then the hash table is filled up and take less than 1mn
Many thanks to look at it when possible, an optimisation would be greatly appreciable on this preallocation part
BTW, on my Core-I5 3570 with native windows 10, this is almost immediate, there is a big gap! I would be tempted to try with virtualbox on my I5 just to see if it's a wine problem or a hardware problem...

Roman · « **Reply #11 on:** 19 March 2018, 10:56 »

no no no...the preallocation should be no time at all...since this is just a one line allocate memory......I thought the filling (during progress bar display) takes long....very strange...I will have a look what happens before the actual progress bar stuff.....(still weird...since nothing really happens there...)

gatti · « **Reply #12 on:** 19 March 2018, 11:09 »

yes it's a very slow memory allocation problem, I'm currently installing a fresh debian on virtualbox on my windows computer to see if the problem is linux/wine or hardware

gatti · « **Reply #13 on:** 19 March 2018, 12:48 »

Same problem under virtualbox on my Windows10 PC with Intel I5, it took 25mn for memory allocation, so it's not an hardware problem
I'll now try 2 other things :
- Wine32/CMPro32
- Debian Stretch (stable) instead of Debian Buster (testing)

Roman · « **Reply #14 on:** 19 March 2018, 13:05 »

I highly doubt it's based on manual allocation....
you shouldn't waste time finding something on your end until I looked at the source to see what's happening there...by the way what's mn stand for? don't say minutes....In the meantime I've tested in on an Ubuntu Virtual box with wine...and I don't see any enormous delay.....

gatti · « **Reply #15 on:** 19 March 2018, 13:12 »

ok I don't say minutes then (but I think hard enough)

gatti · « **Reply #16 on:** 19 March 2018, 15:25 »

One more thing... this memory alloc slow down is happening only with software lists included (with all configured systems/roms paths), it's immediate (<1s) with no software list

Roman · « **Reply #17 on:** 19 March 2018, 16:22 »

now *that* is interesting and actually you should use softwarelists in singe profiles, not in the combined mode, especially not when you're on an old machine.

gatti · « **Reply #18 on:** 20 March 2018, 06:19 »

Indeed, Intel J1800 is a slow processor, so yes, I could use batch mode with single profiles for software lists, *but* with my I5-3570 it's anything but slow, combined mode is very fast under windows, and even with virtualbox+wine overhead, that's not normal that the allocation phase for hash tables goes from <1s on windows to >1500s on VB+Linux+Wine with the same computer...
Anyway, I'll check on a Dual Xeon E5-2640v3 server with 256GB DDR4@2400, SSD only, running Proxmox

Roman · « **Reply #19 on:** 20 March 2018, 07:29 »

use native Windows then. cmpro is not intended to be used under WINE ;-)

News:

Author Topic: Open source on Github (Read 13904 times)