RabbitMQ / AMQP: single queue, multiple consumers for same message?

Question

I am just starting to use RabbitMQ and AMQP in general.

• I have a queue of messages
• I have multiple consumers, which I would like to do different thing

Answer 1

Fan out was clearly what you wanted. fanout

read rabbitMQ tutorial: https://www.rabbitmq.com/tutorials/tutorial-three-javascript.html

here's my example:

Publisher.js:

amqp.connect('amqp://<user>:<pass>@<host>:<port>', async (error0, connection) => {
    if (error0) {
      throw error0;
    }
    console.log('RabbitMQ connected')
    try {
      // Create exchange for queues
      channel = await connection.createChannel()
      await channel.assertExchange(process.env.EXCHANGE_NAME, 'fanout', { durable: false });
      await channel.publish(process.env.EXCHANGE_NAME, '', Buffer.from('msg'))
    } catch(error) {
      console.error(error)
    }
})

Subscriber.js:

amqp.connect('amqp://<user>:<pass>@<host>:<port>', async (error0, connection) => {
    if (error0) {
      throw error0;
    }
    console.log('RabbitMQ connected')
    try {
      // Create/Bind a consumer queue for an exchange broker
      channel = await connection.createChannel()
      await channel.assertExchange(process.env.EXCHANGE_NAME, 'fanout', { durable: false });
      const queue = await channel.assertQueue('', {exclusive: true})
      channel.bindQueue(queue.queue, process.env.EXCHANGE_NAME, '')

      console.log(" [*] Waiting for messages in %s. To exit press CTRL+C");
      channel.consume('', consumeMessage, {noAck: true});
    } catch(error) {
      console.error(error)
    }
});

here is an example i found in the internet. maybe can also help. https://www.codota.com/code/javascript/functions/amqplib/Channel/assertExchange

Answer 2

If you happen to be using the amqplib library as I am, they have a handy example of an implementation of the Publish/Subscribe RabbitMQ tutorial which you might find handy.

Answer 3

There is one interesting option in this scenario I haven`t found in answers here.

You can Nack messages with "requeue" feature in one consumer to process them in another. Generally speaking it is not a right way, but maybe it will be good enough for someone.

https://www.rabbitmq.com/nack.html

And beware of loops (when all concumers nack+requeue message)!

Answer 4

Just read the rabbitmq tutorial. You publish message to exchange, not to queue; it is then routed to appropriate queues. In your case, you should bind separate queue for each consumer. That way, they can consume messages completely independently.

Answer 5

The last couple of answers are almost correct - I have tons of apps that generate messages that need to end up with different consumers so the process is very simple.

If you want multiple consumers to the same message, do the following procedure.

Create multiple queues, one for each app that is to receive the message, in each queue properties, "bind" a routing tag with the amq.direct exchange. Change you publishing app to send to amq.direct and use the routing-tag (not a queue). AMQP will then copy the message into each queue with the same binding. Works like a charm :)

Example: Lets say I have a JSON string I generate, I publish it to the "amq.direct" exchange using the routing tag "new-sales-order", I have a queue for my order_printer app that prints order, I have a queue for my billing system that will send a copy of the order and invoice the client and I have a web archive system where I archive orders for historic/compliance reasons and I have a client web interface where orders are tracked as other info comes in about an order.

So my queues are: order_printer, order_billing, order_archive and order_tracking All have the binding tag "new-sales-order" bound to them, all 4 will get the JSON data.

This is an ideal way to send data without the publishing app knowing or caring about the receiving apps.

Answer 6

As I assess your case is:

• I have a queue of messages (your source for receiving messages, lets name it q111)

• I have multiple consumers, which I would like to do different things with the same message.

Your problem here is while 3 messages are received by this queue, message 1 is consumed by a consumer A, other consumers B and C consumes message 2 and 3. Where as you are in need of a setup where rabbitmq passes on the same copies of all these three messages(1,2,3) to all three connected consumers (A,B,C) simultaneously.

While many configurations can be made to achieve this, a simple way is to use the following two step concept:

• Use a dynamic rabbitmq-shovel to pickup messages from the desired queue(q111) and publish to a fanout exchange (exchange exclusively created and dedicated for this purpose).
• Now re-configure your consumers A,B & C (who were listening to queue(q111)) to listen from this Fanout exchange directly using a exclusive & anonymous queue for each consumer.

Note: While using this concept don't consume directly from the source queue(q111), as messages already consumed wont be shovelled to your Fanout exchange.

If you think this does not satisfies your exact requirement... feel free to post your suggestions :-)