Ruby IMAP IDLE concurrency - how to tackle?

Question

I'm trying to build a (private, for now) web application that will utilize IMAP IDLE connections to show peoples emails as they arrive.

I'm having a hard time figuring out how to hack this together - and how it would fit together with my Heroku RoR server.

I've written a basic script for connecting to an IMAP server and idling, looks something like this (simplified):

imap = Net::IMAP.new server, port, usessl
imap.login username, password
imap.select "INBOX"

imap.add_response_handler do |response|
  if resp.kind_of(Net::IMAP::UntaggedResponse) && resp.name == "EXISTS"
    # New mail recieved. Ping back and process.
  end
end

imap.idle
loop do
  sleep 10*60
  imap.renew_idle
end

This will make one connection to the IMAP server and start idling. As you see, this is blocking with the loop.

I would like to have multiple IMAP connections idling at the same time for my users. Initially, I just wanted to put each of them in a thread, like so:

Thread.new do
  start_imap_idling(server, port, usessl, username, password)
end

I'm not that sharp on threads yet, but with this solution I will still have to block my main thread to wait for the threads? So if I do something like:

User.each do |user|
  Thread.new do
    start_imap_idling(server, port, usessl, username, password)
  end
end

loop do
  # Wait
end

That would work, but not without the loop at the bottom to allow the threads to run?

My question is how I best melt this together with my Ruby On Rails application on Heroku? I can't be blocking the thread with that last loop - so how do I run this? Another server? A dyno more - perhaps a worker? I've been reading a bit about Event Machine - could this solve my problem, if so, how should I go about writing this?

Another thing is, that I would like to be able to add new imap clients and remove current ones on the fly. How might that look? Something with a queue perhaps?

Any help and comments are very much appreciated!

Answer 1

I'm not familiar with the specifics of RoR, Event Machine, etc. -- but it seems like you'd want to set up a producer/consumer.

The producer is your thread that's listening for changes from the IMAP server. When it gets changes it writes them to a queue. It seems like you'd want to set up multiple producers, one for each IMAP connection.

Your consumer is a thread that blocks on read from the queue. When something enters the queue it unblocks and processes the event.

Your main thread would then be free to do whatever you want. It sounds like you'd want your main thread doing things like adding new IMAP clients (i.e., producers) and removing current ones on the fly.

As for where you'd run these things: You could run the consumers and producer in one executable, in separate executables on the same machine, or on different machines... all depending upon your circumstances.

HTH