RabbitMQ plugin to remove duplicate messages

Question

I have a RabbitMQ queues for documents generation. Basically, each document has type and state (new, processing, ready), so I use topic exchange with r

Answer 1

You can check this plugin I wrote which allows to de-duplicate messages published within the broker.

You can de-duplicate on the exchange or at the queue according to your needs. Only thing your publisher needs to do is to set the x-deduplicate-message message header with the ID of your message.

Answer 2

As you wrote, ActiveMQ has "duplicate message detection", but it works differently. It does not remove old message from the queue but it does not add new message to it instead. So it works the same as plugin for RabbitMQ.

Answer 3

Ok, i've read about RabbitMQ inner architecture and find out it's impossible. So the way around for somebody looking for it.

1. Send only document ID in message body
2. Create a key-value store for worker (i use memcached for this). Key is ID value is timestamp of last worker run for this ID.
3. When worker receives the message it checks if message timestamp greater then the one from key-value store. If it is, then update timestamp in the store and run the task, otherwise just skip it.