RabbitMQ: fast producer and slow consumer

Question

I have an application that uses RabbitMQ as the message queue to send/receive message between two components: sender and receiver. The sender sends message in a very fast way. T

Answer 1

While it is true adding more consumers may speed things up the real issue will be saving to the database.

There are already many answers here that talk about adding consumers (threads, and or machines) and changing the QoS so I'm not going to reiterate that. Instead you should seriously consider using the Aggregator pattern to aggregate the messages into a group of messages and then batch insert the group into your database in one shot.

Your current code for each message probably opens up a connection, inserts the data, and the closes that connection (or return to the pool). Worse it may even be using transactions.

By using the aggregator pattern your essentially buffering the data before you flush.

Now writing a good aggregator is tricky. You will need to decide how you want to buffer (ie each worker has its own buffer or a central buffer like Redis). Spring integration has an aggregator I believe.