Python: writing large array of arrays to text file

问题

I'm new to Python and I have a solution for this but it seems slow and silly, so I wondered if there is a better way?

Say I have a matrix defined like this:

mat = [['hello']*4 for x in xrange(3)]

I am using this function to write it to file:

def writeMat(mat, outfile):
  with open(outfile, "w") as f:
    for item in mat:
      f.writelines(str(item).replace('[','').replace(',','').replace('\'','').replace(']','\n'))

writeMat(mat, "temp.txt")

which gives a text file that looks like:

hello hello hello hello
hello hello hello hello
hello hello hello hello

The files that I am dealing with are very large. The savetxt function in numpy would be great, but I don't want to store this as a numpy array because while the majority of the matrix is comprised of single character elements, the first few columns will be many characters in length, and it seems to me (correct me if I am wrong) this would mean the whole matrix would use much more memory than is necessary because every element in the matrix will be the size of the largest element.

回答1:

If I understand your question correctly, you could do:

f.writelines(' '.join(row) + '\n' for row in mat)

f.write('\n'.join(' '.join(row) for row in mat))

The first one has the advantage of being a generator expression that only makes a concatenated string copy of the currentline

And if your matrix entries are not strings, you could do:

f.writelines(' '.join(str(elem) for elem in row) + '\n' for row in mat)

EDIT

It appears that the file.writelines() method evaluates the entire generator expression before writing it to the file. So the following would minimize your memory consumption:

for row in mat:
    f.write(' '.join(row) + '\n')

回答2:

You could use csv module:

import csv

with open(outfile, 'wb') as f:
     csv.writer(f, delimiter=' ').writerows(mat)

来源：https://stackoverflow.com/questions/11589583/python-writing-large-array-of-arrays-to-text-file

标签

python

matrix

text-files