How to prevent duplicate SQS Messages?

Question

What is the best way to prevent duplicate messages in Amazon SQS? I have a SQS of domains waiting to be crawled. before I add a new domain to the SQS I can check with the saved

Answer 1

As you can't prevent SQS from sending the duplicated messages, you must implement this on your side. One simple way to do this is using Apache Camel's idempotent consumers, see http://camel.apache.org/idempotent-consumer.html