Many small files or one big file? (Or, Overhead of opening and closing file handles) (C++)

Question

I have created an application that does the following:

1. Make some calculations, write calculated data to a file - repeat for 500,000 times (over al

Answer 1

Opening a file handle isn't probable to be the bottleneck; actual disk IO is. If you can parallelize disk access (by e.g. using multiple disks, faster disks, a RAM disk, ...) you may benefit way more. Also, be sure to have IO not block the application: read from disk, and process while waiting for IO. E.g. with a reader and a processor thread.

Another thing: if the next step depends on the current calculation, why go through the effort of saving it to disk? Maybe with another view on the process' dependencies you can rework the data flow and get rid of a lot of IO.

Oh yes, and measure it :)