RabbitMQ: fast producer and slow consumer

Question

I have an application that uses RabbitMQ as the message queue to send/receive message between two components: sender and receiver. The sender sends message in a very fast way. T

Answer 1

While it is true adding more consumers may speed things up the real issue will be saving to the database.

There are already many answers here that talk about adding consumers (threads, and or machines) and changing the QoS so I'm not going to reiterate that. Instead you should seriously consider using the Aggregator pattern to aggregate the messages into a group of messages and then batch insert the group into your database in one shot.

Your current code for each message probably opens up a connection, inserts the data, and the closes that connection (or return to the pool). Worse it may even be using transactions.

By using the aggregator pattern your essentially buffering the data before you flush.

Now writing a good aggregator is tricky. You will need to decide how you want to buffer (ie each worker has its own buffer or a central buffer like Redis). Spring integration has an aggregator I believe.

Answer 2

You have lot of ways to increase your performance.

1. You can create a worker queue with more producers, in this way you create an simple load-balance system. don't use exchange---> queue but only queue. Read this post RabbitMQ Non-Round Robin Dispatching

2. When you get a message you can create a poolthread for insert the data on your database, but in this case you have to manage the failure.

But I think the principal problem is the database and not RabbitMQ. With a good tuning, multi-threading and worker queue you can have a scalable and fast solution.

Let me know

Answer 3

As answer I suggest: both.

You can take advantage from having multiple receiver, as well as setting up each receiver to execute the task in a separate Thread, thus permitting to the receiver to accept the next message in queue.

Of course this approach assumes that the result of each operation (the writing on the db, if I understood correctly) does not influence in any way the result of the subsequent operations in response from other messages.

Answer 4

"So how can I speed up the consumer throughput so that the consumer can catch up with the producer and avoid the message overflow in the queue?" This is the answer "use multiple consumers to consume the incoming message simultaneously", use multi-threading to run in parallel these consumers implementing principle shared nothing, http://www.eaipatterns.com/CompetingConsumers.html

Answer 5

"Will this cause the message queue to overflow?"

Yes. RabbitMQ will enter a state of "flow control" to prevent excessive memory consumption as the queue length increases. It will also start persisting messages to disk, rather than hold them in memory.

"So how can I speed up the consumer throughput so that the consumer can catch up with the producer and avoid the message overflow in the queue"

You have 2 options:

1. Add more consumers. Bear in mind that your DB will now be manipulated by multiple concurrent processes if you choose this option. Ensure that the DB can withstand the extra pressure.
2. Increase the QOS value of the consuming channel. This will pull more messages from the queue and buffer them on the consumer. This will increase the overall processing time; if 5 messages are buffered, the 5th message will take the processing time of messages 1...5 to complete.

"Should I use multithreading in the consumer part to speed up the consumption rate?"

Not unless you have a well-designed solution. Adding parallelism to an application is going to add a lot of overhead on the consumer-side. You may end up exhausting the ThreadPool or throttling memory-usage.

When dealing with AMQP, you really need to consider the business requirement for each process in order to design the optimal solution. How time-sensitive are your incoming messages? Do they need to be persisted to DB ASAP, or does it matter to your users whether or not that data is available immediately?

If the data does not need to be persisted immediately, you could modify your application so that the consumer(s) simply remove messages from the queue and save them to a cached collection, in Redis, for example. Introduce a second process which then reads and processes the cached messages sequentially. This will ensure that your queue-length does not grow sufficiently to result in flow-control, while preventing your DB from being bombarded with write requests, which are typically more expensive than read requests. Your consumer(s) now simply remove messages from the queue, to be dealt with by another process later.