redis memory and cpu spikes

Question

We use redis for some data in our app, and it\'s totally great. I noticed however occasional cpu and memory spikes on the redis-server process.

Answer 1

The docs says: "the Redis AOF works incrementally updating an existing state, like MySQL or MongoDB does, while the RDB snapshotting creates everything from scratch again and again, that is conceptually more robust."

Source: http://redis.io/topics/persistence (in AOF disadvantages)

Answer 2

From experimenting further with this and reading about redis persistence, I think the following observations can be made:

• When using RDB (the default settings), redis will fork each time a save operation is triggered, which (by default) is set to once every 15 minutes as a minimum. When more writes to Redis are performed, then RDB writes are as frequent as once every 60 seconds.
• Each fork will use a "copy-on-write" memory allocation, which means that whilst memory won't actually double - it will appear so on tools like ps, htop and the like.
• The fork itself can be quite a cpu-intensive operation, particularly on xen-based virtual hosts (which is what we're using currently).
• The write operation seems to completely overwrite the existing RDB file. It does not write only the changes, but rather dumps the entire dataset to disk.

So on a modest virtual host with 4Gb RAM and data set of around 750Mb (at the time I posted the question), this starts to become rather "expensive". We observed those CPU/Memory spikes, as well as increased IO, even under fairly moderate load / redis usage.

So to answer my own question - this does seem to be the "expected" behaviour.

As for improving the situation, we opted to switch our configuration to use a combination of RDB and AOF. AOF (Append Only File), does appear to only write changes to disk. You can (and should) still configure the AOF file to rewrite (using auto-aof-rewrite-percentage and auto-aof-rewrite-min-size settings). It is also advisable to still use RDB for snapshots. In this configuration however, you can probably do full rewrites / snapshots less frequently and still maintain pretty-good performance and even-better durability.

Answer 3

If I recall correctly, redis forks the process when it does a background save, but only duplicates the memory that is being changed while the save is in progress. So the bump in CPU/memory would depend heavily on how much of the data is being changed while the save is running. So it could certainly be a huge spike at times, and a much lesser spike other times (or none at all, depending on how your load looks).