How can I save a very large MATLAB sparse matrix to a text file?

Question

I have a 30000x14000 sparse matrix in MATLAB (version 7), which I need to use in another program. Calling save won\'t write this as ASCII (not supported). Calling full()

Answer 1

dlmwrite - Write matrix to ASCII-delimited file Syntax

dlmwrite(filename, M)

dlmwrite(filename, M, 'D')

dlmwrite(filename, M, 'D', R, C)

dlmwrite(filename, M, 'attrib1', value1, 'attrib2', value2, ...)

dlmwrite(filename, M, '-append')

dlmwrite(filename, M, '-append', attribute-value list)

Answer 2

I saved it as text using Java within MATLAB. MATLAB Code:

pw=java.io.PrintWriter(java.io.FileWriter('c:\\retail.txt'));
line=num2str(0:size(data,2)-1);
pw.println(line);
for index=1:length(data)
    disp(index);
    line=num2str(full(data(index,:)));
    pw.println(line);
end
pw.flush();
pw.close();

Here data is an extremely large sparse matrix.

Answer 3

You can use find to get index & value vectors:

[i,j,val] = find(data)
data_dump = [i,j,val]

You can recreate data from data_dump with spconvert, which is meant to "Import from sparse matrix external format" (so I guess it's a good export format):

data = spconvert( data_dump )

You can save to ascii with:

save -ascii data.txt data_dump

But this dumps indices as double, you can write it out more nicely with fopen/fprintf/fclose:

fid = fopen('data.txt','w')
fprintf( fid,'%d %d %f\n', transpose(data_dump) )
fclose(fid)

Hope this helps.

Answer 4

Use the find function to get the indices of non-zero elements...

idcs = find(data);
vals = data(idcs);
...save the index vector and value vector in whatever format you want...

If you want, you can use ind2sub to convert the linear indices to row, column subscripts.

If you need to recreate a sparse matrix in matlab from subscripts + values, use spconvert.

Answer 5

Did you try partitioning it ?

I mean try calling full() on the 1000 first rows (or 5000) and then repeat the process if it works.

Answer 6

Save the sparse matrix as a .mat file. Then, in the other program, use a suitable library to read the .mat file.

For instance, if the other program is written in Python, you can use the scipy.io.mio.loadmat function, which supports sparse arrays and gives you a sparse numpy matrix.