unbuffered I/O in Linux

Question

I\'m writing lots and lots of data that will not be read again for weeks - as my program runs the amount of free memory on the machine (displayed with \'free\' or \'top\') drops

Answer 1

You can use O_DIRECT, but in that case you need to do the block IO yourself; you must write in multiples of the FS block size and on block boundaries (it is possible that it is not mandatory but if you do not its performance will suck x1000 because every unaligned write will need a read first).

Another much less impacting way of stopping your blocks using up the OS cache without using O_DIRECT, is to use posix_fadvise(fd, offset,len, POSIX_FADV_DONTNEED). Under Linux 2.6 kernels which support it, this immediately discards (clean) blocks from the cache. Of course you need to use fdatasync() or such like first, otherwise the blocks may still be dirty and hence won't be cleared from the cache.

It is probably a bad idea of fdatasync() and posix_fadvise( ... POSIX_FADV_DONTNEED) after every write, but instead wait until you've done a reasonable amount (50M, 100M maybe).

So in short

• after every (significant chunk) of writes,
• Call fdatasync followed by posix_fadvise( ... POSIX_FADV_DONTNEED)
• This will flush the data to disc and immediately remove them from the OS cache, leaving space for more important things.

Some users have found that things like fast-growing log files can easily blow "more useful" stuff out of the disc cache, which reduces cache hits a lot on a box which needs to have a lot of read cache, but also writes logs quickly. This is the main motivation for this feature.

However, like any optimisation

a) You're not going to need it so

b) Do not do it (yet)