Optimizing disk IO

Question

I have a piece of code that analyzes streams of data from very large (10-100GB) binary files. It works well, so it\'s time to start optimizing, and currently disk IO is the bigg

Answer 1

First off, it would be useful to use a profiler to identify the hot spots in your program. Based on your description of the problem, you have a lot of overhead going on by the sheer number of freads. As the files are large there will be a big benefit to increasing the amount of data you read per io.

Convince yourself of this by putting together 2 small programs that read the stream.

1) read it as you are in the example above, of 2 doubles.

2) read it the same way, but make it 10,000 doubles.

Time both runs a few times, and odds are you will be observe #2 runs much faster.

Best of luck.