Message Queues in Ruby on Rails

Question

What message queues are people using for their Rails apps and what was the driving force behind the decision to choose it. Does the latest Twitter publicity over their in ho

Answer 1

Rany Keddo gave a useful presentation about Starling + Workling at RailsConf Europe last year. He compared the different solutions available at the time.

Twitter's latest move away from Starling + Workling probably doesn't mean much to the regular rails app. They have a lot more issues of scale and probably have legacy issues with their datastore that prevents them from scaling past their current implementation.

Beanstalkd is a good alternative, simply because it runs as a daemon and has wrappers in other scripting languages (if you happen to change direction in the future or have different components written in other languages).

This link also has a good comparison of pros-cons of the various rails solutions available.

Answer 2

Here are a few Ruby/Rails solutions, one or more of these may be a good fit depending on your needs:

http://xph.us/software/beanstalkd

http://rubyforge.org/forum/forum.php?forum_id=19781

http://backgroundrb.rubyforge.org

And, a hosted solution from Amazon which would make a great queue for sharing between Ruby/Rails and other components of a larger system:

http://aws.amazon.com/sqs

Hope this helps!

Answer 3

As an update -- GitHub have moved to Resque on Redis instead of Delayed job. However they still recommend delayed_job for smaller setups:

https://github.com/resque/resque

Answer 4

I would recommend delayed-job as a dead simple solution if you don't expect any heavy load. Pros: easy to setup, easy to monitor, simple code, doesn't have any external dependencies. Previously we used ActiveMessaging (with ActiveMQ and stomp), but it was an overkill for our project, so we switched to delayed_job for its simplicity.

Anyway, if you need very mature and fast solution, ActiveMQ is a very good choice. If you don't want to spend too much time on maintaining full-scale message queueing solution you don't really need, delayed_job is a way to go. Here's a good article about Scribd experience with ActiveMQ.

Answer 5

Chris Wanstrath from github was at the SF Ruby meetup recently, talking about their queue. They tried Starling, beanstalk, and some other variants before settling on Shopify's delayed_job. They are pretty aggressive with their use of backgrounding.

Here's a blog post from last year that talks about their move to DJ.

Where I am now we rolled our own several years ago, but I'm taking some ideas from DJ to improve the handling.

Answer 6

The Messaging Server you might want to go for is RabbitMQ. Erlang coolness, AMQP, good Ruby libs.

http://www.bestechvideos.com/2008/12/09/rabbitmq-an-open-source-messaging-broker-that-just-works