what is the fastest way in python to convert a string with formatted numbers in an numpy array

Question

I have a large ASCII file (~100GB) which consists of roughly 1.000.000 lines of known formatted numbers which I try to process with python. The file is too large to read in comp

Answer 1

Have you tried numpyp.fromstring?

np.fromstring(line, dtype=np.float, sep=" ")

Answer 2

The np.genfromtxt function is a speed champion if you can get it to match you input format.

If not, then you may already be using the fastest method. Your line-by-line split-into-array approach exactly matches the SciPy Cookbook examples.