What is the fastest way to write a matrix to a text file in Octave?

问题

I have a large matrix (2e6 x 3) which I have to write to a text file.

dlmwrite takes about 230s to achieve this task.

From your experience what is the fastest way to write a large matrix to a text file?

回答1:

The following applies to MATLAB, but I suggest you try it in Octave. First of all, if you can - transpose the matrix. Here are examples using fprintf and csvwrite (essentially dlmwrite)

A = rand(3, 1e6);
tic;
fid = fopen('data.txt', 'w+');
for i=1:size(A, 1)
    fprintf(fid, '%f ', A(i,:));
    fprintf(fid, '\n');
end
fclose(fid);
toc

tic;
csvwrite('data.txt', A);
toc;

Elapsed time is 1.311512 seconds.
Elapsed time is 2.487737 seconds.

If not transposed, it will take ages indeed. By default, fprintf flushes the buffer after every call. You can try to use W instead of w to open the file, but it does not improve the situation here too much.

回答2:

Did you try this? I'm not sure about it's speed as compared to dlmwrite.

a = [1 2;3 4];
save temp.txt a;

回答3:

Having a variable data you can save it in a text format with space separated values (including a header):

save out.txt data

The header can be simply removed using basic Unix command tail e.g. (on any Linux/Mac OS):

tail -n +6 out.txt > file.csv

回答4:

Theoretically, according to what @angainor says, one can even improve performance wrapping somehow

for i=1:size(A, 1)
   fprintf(fid, '%f ', A(i,:));
   fprintf(fid, '\n');
end

in chunks to avoid useless buffer flushing, i.e.doing

1. coverting (in memory) numbers->string + introduction of termination characters '\n' 
   (for say K matrix rows)
2. writing of the obtained string to file through fscanf.

Ought to try..

回答5:

Use this in Matlab:

 save -ascii output.txt variableName

Use this in Octave:

 save hello1.m variableName -ascii

回答6:

In my system

A = rand(3, 1e6);

# Method with fprintf
tic;
fid = fopen('data1.txt', 'w+');
for i=1:size(A, 1)
    fprintf(fid, '%f ', A(i,:));
    fprintf(fid, '\n');
end
fclose(fid);
toc

# Method with sprintf
tic;
s = "";
for i=1:size(A, 1)
    s = strcat( s, sprintf('%f ', A(i,:)) );
    s = strcat( s, sprintf('\n') );
end
fid = fopen('data2.txt', 'w+');
    fprintf(fid, '%s\n', s);
fclose(fid);
toc

# Method with save
tic;
save 'data3.txt' A;
toc;

return;  # Commented when the size is <= 1e5

# Method with csvwrite
tic;
csvwrite('data4.txt', A);
toc;

gives

>> Elapsed time is 5.36293 seconds.
Elapsed time is 6.43252 seconds.
Elapsed time is 6.09889 seconds.

Since csvwrite is about 10 times slower than the others, I only tried it with a size = 10^-5. In that case,

>> Elapsed time is 0.541885 seconds.
Elapsed time is 0.657595 seconds.
Elapsed time is 0.576796 seconds.
Elapsed time is 4.24433 seconds.

My conclusions are:

The comparison in speed among the various methods is strongly dependent on the system. Then, you will have to try yourself.
The proposal by Acorbe does not meet his expectations.

来源：https://stackoverflow.com/questions/12932052/what-is-the-fastest-way-to-write-a-matrix-to-a-text-file-in-octave

标签

file

matlab

matrix

octave