Redis is slow to get large strings

Question

I\'m kind of a newb with Redis, so I apologize if this is a stupid question.

I\'m using Django with Redis as a cache.

I\'m pickling a collection of ~200 objects

Answer 1

It's most likely just the size of the string. I'd look at whether your objects are being serialized efficiently.

Answer 2

Redis is not designed to store very large objects. You are not supposed to store your entire collection in a single string in Redis, but rather use Redis list or set as a container for your objects.

Furthermore, the pickle format is not optimized for space ... you would need a more compact format. Protocol Buffers, MessagePack, or even plain JSON, are probably better for this. You should consider to apply a light compression algorithm before storing your data (like Snappy, LZO, Quicklz, LZF, etc ...).

Finally, the performance is probably network bound. On my machine, retrieving a 20 MB object from Redis takes 85 ms (not 3 seconds). Now, if I run the same test using a remote server, it takes 1.781 seconds, which is expected on this 100 Mbit/s network. The duration is fully dependent on the network bandwidth.

Last point: be sure to use a recent Redis version - a number of optimization have been done to deal with large objects.