Job queue with redis using BLPOP

Question

Im trying to create infinite job queue using redis and ruby eventmachine. To achieve that Im using redis BLPOP command with 0 timeout. After successful BLPOP I run it again.

Answer 1

If you use BLPOP alone to remove a message from the queue, and your message consumer fails to process it, the message will have to be re-queued, lest it disappear forever along with the failed consumer.

For more durable message processing, a list of messages being processed must be maintained so they can be re-queued in the event of failure.

[B]RPOPLPUSH is perfect for this scenario; it can atomically pop a message from the message queue and pushes it onto a processing queue so that the application can respond in the case of a failure on the consumer's end.

http://redis.io/commands/rpoplpush

Actual re-queueing is left to the application, but this redis command provides the foundations to do so.

There are also some drop-in-place implementations of queues using redis floating around the web, such as RestMQ [ http://www.restmq.com/ ]