Why would saving to a folder called temp cause data loading to slow down in a for loop in Matlab?

Question

IMPORTANT UPDATE

I just made the discovery that after restarting Matlab and the computer, this simplified code no longer reproduces the problem for me either... I am so

Answer 1

I can't reproduce the problem, suspect it's system and data-size specific. But some general comments which could help you out of the predicament:

As pointed out by commenters and the above answers, file i/o within a double for loop can be extremely parasitic, especially in cases where you only need to access part of the data in the file, where other system operations delay the process, or where the data files are large enough to require virtual memory (windows) / swap space (linux) to even load them. In the latter case, you could be in a situation where you're moving a file from one part of the hard disk to another when you open it!

I assume that you're loading/saving because you don't have c.10GB of ram to hold everything in memory for computation. The actual problem is not described, so I can't be certain, but think you might find that the matfile class to be useful... TMW documentation. This is used to map directly to/from a mat file. This:

• reduces file stream opening and closing IOPS

• allows arbitrarily large variable sizes (governed by disk size, not memory)

• allows you to read/write partially (i.e. write only some elements of an array without loading the whole file)

• in the case that your mat file is too large to be held in memory, avoids loading it into swap space which would be extremely cumbersome.

Hope this helps.

Tom

Answer 2

There are two things here

1. Storage during a for loop is an expensive operation as it usually opens a file stream and closes it before it moves on. You might not be able to avoid this.
2. Second thing is speed of storage and its cache speed. Most likely programs use temp folder for its own temporary files and have a garbage collector or software looking after these to clean them. If you start opening and closing file stream to this folder you have to send a request to get exclusive write access to the folder. This again adds to the time.

If you are doing image processing operations and you have multiple images you can run into a bottle neck with writing to hard drive due to its speed, cache and current memory available to MATLAB.