Interesting times playing with Linux ... am like a kid with a new bike.
This is scalable
a) using one subset of a 10Gig project - standardising the data (as text) files to make readable for Linux went from 800MB to 300MB without loss of information. All in windoze.
b) searching that subset data on windoze took 29.7 secs
c) the exact same search of the exact same data on Linux took 1.2 sec (one point two)
d) using the most optimal windoze routine I can get it down to 15.5 secs - still 12x slower
Simple math. So what this means is on Linux in less 30 seconds with 1 thread I can do more than 18 concurrent threads on windows. I can search 3.6GB of data with 1 thread in Linux in less than the time windoze needs 12 threads.
Mind(blown). :boom:
{edit} The best it gets for windoze is a single thread sprint of an identical routine on the same 3.0Gig data structure spread across 9 folders. Linux done in 24.7secs retrieved 3.287M results in a 528 MB file. Windoze 1min 26sec. retrieved 3.283M results in a 481 MB file.
Hey ... windoze maybe slow but at least it's not accurate.
It has been an interesting introduction to Linux. I have learned a lot about the intentional misgivings of windows not throwing exceptions to poorly formed data just to keep me happy. Sadly this means wasted space in not forcing me to keep house properly. Standardising to Linux found 60% saving in file size without loss of data before compression. The latest side by side result shows 3min 05s to do what Linux does in 1min 05s with matching results (finally!!).
I figure using windows is like giving smarties to kids. Sure it makes them happy for a while but it doesn't make them better people. My 20c is up.