How to prevent duplicate SQS Messages?

Question

What is the best way to prevent duplicate messages in Amazon SQS? I have a SQS of domains waiting to be crawled. before I add a new domain to the SQS I can check with the saved

Answer 1

You can use the VisibilityTimeout parameter:

var params = { 
    VisibilityTimeout: 20,
    ...
};

sqs.receiveMessage(params, function(err, data) {});

reference: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

VisibilityTimeout — (Integer)
The duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request.