问题
A month ago, out of the blue, Redis started to fill up the server memory fast. In order to debug the problem we have run redis-cli --bigkeys and for our surprise all the used memory was freed.
We have a cluster of 6 nodes, being 3 masters and 3 slaves, each of the masters databases are around 15GB. Each of the nodes are stored in a dedicated box with 64GB each. Redis is filling the entire memory of 64GB twice a day. We have a cron running redis-cli --bigkeys twice a day to free up the used memory.
What could be the cause?
Thank you.
回答1:
Sounds like you are getting OOM command not allowed errors unless you run redis-cli --bigkeys twice a day.
If that's the case, you probably have many and/or big keys with EXPIRE, being added constantly. Expired keys are removed from memory:
- Passively: when you try to access it and the key is found to be timed out. This is how
redis-cli --bigkeysis helping you, it forces a passive removal across all the keyspace. - Actively: every 100 ms, it tries to remove from memory expired keys at random, never investing more than 1 ms per cycle at it, until it estimates that less than 25% of expired keys remain. The logic is not that trivial, see activeExpireCycle.
So it all points that the active expire is not able to catch up in your case.
From your comment, maxmemory=0 and maxmemory-policy=noeviction. You may want to consider setting a maxmemory value, and maxmemory-policy=noeviction to volatile-ttl (remove the key with the nearest expire time).
What this does, whenever a write commands finds you are over maxmemory, it will try to free space for the new key, based on the policy. The volatile-ttl policy would evict first any expired keys remaining. See evict.c.
You may also increase the frequency of the background tasks, to purge expired keys more often, see hz in redis.conf. You may double it to 20.
By default "hz" is set to 10. Raising the value will use more CPU when Redis is idle, but at the same time will make Redis more responsive when there are many keys expiring at the same time, and timeouts may be handled with more precision.
Also, activedefrag = yes may help, see here.
There is a new active-expire-effort redis.conf setting that would allow you to invest more CPU in active-expire, but it is not available in the latest stable release (5.0.7).
Use INFO memory to get a sense of your redis server memory status. Please update the question with this output if the above doesn't help you.
来源：https://stackoverflow.com/questions/59777591/redis-filling-up-memory-fast-running-bigkeys-free-it-up