Using Celery with existing RabbitMQ messages

Question

I have an existing RabbitMQ deployment that that a few Java applications are using the send out log messages as string JSON objects on various channels. I would like to use Cel

Answer 1

It's currently hard to add custom consumers to the celery workers, but this is changing in the development version (to become 3.1) where I've added support for Consumer boot-steps.

There's no documentation yet as I've just finished implementing it, but here's an example:

from celery import Celery
from celery.bin import Option
from celery.bootsteps import ConsumerStep
from kombu import Consumer, Exchange, Queue

class CustomConsumer(ConsumerStep):
   queue = Queue('custom', Exchange('custom'), routing_key='custom')

   def __init__(self, c, enable_custom_consumer=False, **kwargs):
       self.enable = self.enable_custom_consumer

   def get_consumers(self, connection):
       return [
           Consumer(connection.channel(),
               queues=[self.queue],
               callbacks=[self.on_message]),
       ]

   def on_message(self, body, message):
       print('GOT MESSAGE: %r' % (body, ))
       message.ack()


celery = Celery(broker='amqp://localhost//')
celery.steps['consumer'].add(CustomConsumer)
celery.user_options['worker'].add(
    Option('--enable-custom-consumer', action='store_true',
           help='Enable our custom consumer.'),
)

Note that the API may change in the final version, one thing that I'm not yet sure about is how channels are handled after get_consumer(connection). Currently the channel of the consumer is closed when connection is lost, and at shutdown, but people may want to handle channels manually. In that case there's always the possibility of customizing ConsumerStep, or writing a new StartStopStep.