How to purge disk I/O caches on Linux?

Question

I need to do it for more predictable benchmarking.

Answer 1

You can do it like this:

# sync # (move data, modified through FS -> HDD cache) + flush HDD cache
# echo 3 > /proc/sys/vm/drop_caches # (slab + pagecache) -> HDD (https://www.kernel.org/doc/Documentation/sysctl/vm.txt)
# blockdev --flushbufs /dev/sda
# hdparm -F /dev/sda

# NEXT COMMAND IS NOT FOR BENCHMARKING:
# should be run before unplug, flushes everything possible guaranteed.
# echo 1 > /sys/block/sdX/device/delete

You may use strace to see that these are three different syscalls

Also, it may be desirable to turn off HDD cache using hdparm, not sure what thing you benchmarking.

In any way, you cannot prevent HDD to cache last 64/32/16 MB of recently used data. In order to kill that cache, just write some amount of zeroes (and flush) + read some unrelated place from HDD. This is required since cache may be divided to read-part and write-part. After that you can benchmark HDD.

Answer 2

Disk cache purging: echo 3 | sudo tee /proc/sys/vm/drop_caches

Command documentation: https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.

To free pagecache:

echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation, and dirty objects are not freeable, the user should run "sync" first in order to make sure all cached objects are freed.

Answer 3

Sounds like you want the sync command, or the sync() function.

If you want disk cache flushing: echo 3 | sudo tee /proc/sys/vm/drop_caches

Answer 4

Short good enough answer: (copy paste friendly)

DISK=/dev/sdX # <===ADJUST THIS===
sync
echo 3 > /proc/sys/vm/drop_caches
blockdev --flushbufs $DISK
hdparm -F $DISK

Explanation:

sync: From the man page: flush file system buffers. Force changed blocks to disk, update the super block.

echo 3 > /proc/sys/vm/drop_cache: from the kernel docs this will cause the kernel to drop clean caches

blockdev --flushbufs /dev/sda: from the man page: call block device ioctls [to] flush buffers.

hdparm -F /dev/sda: from the man page: Flush the on-drive write cache buffer (older drives may not implement this)

Although the blockdev and hdparm commands look similar according to an answer above they issue different ioctls to the device.

Long probably better way:

(I'll assume that you have formatted the disk but you can adapt these commands if you want to write directly to the disk)

Run this only once before the 1st benchmark:

MOUNT=/mnt/test # <===ADJUST THIS===
# create a file with psuedo-random data. We will read it
# to fill the read cache of the HDD with garbage
dd if=/dev/urandom of=$MOUNT/temp-hddread.tmp bs=64M count=16

Run this every time you want to empty the caches:

DISK=/dev/sdX # <===ADJUST THIS===
MOUNT=/mnt/test # <===AND THIS===
# create a file with psuedo-random data to fill the write cache
# of the disk with garbage. Delete it afterwards it's not useful anymore
dd if=/dev/urandom of=$MOUNT/temp-hddwrite.tmp bs=64M count=16
rm $MOUNT/temp-hddwrite.tmp
# see short good enough answer above
sync
echo 3 > /proc/sys/vm/drop_caches
blockdev --flushbufs $DISK
hdparm -F $DISK
# read the file with pseudo-random data to fill any read-cache
# the disk may have with garbage
dd if=$MOUNT/temp-hddread.tmp of=/dev/null

Run this when you're done.

MOUNT=/mnt/test # <===ADJUST THIS===
# delete the temporary file with pseudo-random data
rm $MOUNT/temp-hddread.tmp

Explanation:

The disk will probably have some H/W cache. Some disks by design or due to bugs may not clear their caches when you issue the blockdev and hdparm commands. To compensate we write and read pseudo-random data hopping to fill these caches so that any cached data are removed from them. How much data you need to fill the cache depends on its size. In the commands above I'm using dd to read/write 16*64MB=1024MB, adjust the arguments if your HDD may have bigger cache (data sheets and experimentation are your friend and it doesn't hurt to specify values above the actual size of the cache). I'm using /dev/urandom as a source for random data because it's fast and we don't care about true randomness (we only care for high entropy because the disk firmware may be using compression before storing data to the cache). I'm creating /mnt/test/temp-hddread.tmp from the start and use it every time I want to read enough random data. I'm creating and deleting /mnt/test/temp-hddwrite.tmp each time I want to write enough random data.

Credits

I've wrote this answer based on the best parts of the existing answers.

Answer 5

Unmounting and re-mounting the disk under test will reset all caches and buffers.