Massive CSV file into Matlab

Question

I have a CSV file 1.6 GB large, that I need to feed into matlab. I will have to do this frequently and I need it to run quickly. The file is of the form:

201         


        

Answer 1

The recommended syntax is textscan (http://www.mathworks.com/help/matlab/ref/textscan.html)

Your code would look like this:

fid = fopen('C:\Program Files\MATLAB\R2013a\EDU13.csv','r');
c = textscan(fid, '%d,%d:%d.%d,%f,%d,%c');
fclose(fid);

You end up with a cell array... whether it's worth converting that to another shape really depends on how you want to access the data afterwards.

It is quite likely that this would be faster if you include a loop that allows you to use a smaller, fixed amount of memory for much of the operation. One problem with reading large files is the fact that you don't know ahead of time how big it will be - and that very likely means that Matlab guesses the amount of memory it needs, and frequently has to rescale. That is a very slow operation - if it happens every 1MB, say, then it copies 1 MB once, next 2 MB, then again 3 MB, etc - as you can see it is quadratic in the size of the array.

If instead you allocate a fixed amount of memory for the final result, and process in smaller batches, you avoid all that overhead. I'm pretty sure it will be much faster - but you would have to experiment a bit with the block size. That would look something like this:

block = 1000;
Nlines = 35E6; 
fid = fopen('C:\Program Files\MATLAB\R2013a\EDU13.csv','r');
c = struct(field1, field2, fieldn, value); %... initialize structure array or other storage for c ...
c_offset = 0;
while ~feof(fid)
  temp = textscan(fid, '%d,%d:%d.%d,%f,%d,%c', block);
    bt = size(temp, 1); % first dimension - should be `block`, except for last loop
    %... extract, process, store in c(c_offset + (1:bt))... 
    c_offset = c_offset + bt;
end
fclose(fid);