Retrieve multiple messages from SQS

Question

I have multiple messages in SQS. The following code always returns only one, even if there are dozens visible (not in flight). setMaxNumberOfMessages I th

Answer 1

There is a comprehensive explanation for this (arguably rather idiosyncratic) behaviour in the SQS reference documentation.

SQS stores copies of messages on multiple servers and receive message requests are made to these servers with one of two possible strategies,

• Short Polling : The default behaviour, only a subset of the servers (based on a weighted random distribution) are queried.
• Long Polling : Enabled by setting the WaitTimeSeconds attribute to a non-zero value, all of the servers are queried.

In practice, for my limited tests, I always seem to get one message with short polling just as you did.

Answer 2

I had the same problem. What is your Receive Message Wait Time for your queue set to? When mine was at 0, it only returned 1 message even if there were 8 in the queue. When I increased the Receive Message Wait Time, then I got all of them. Seems kind of buggy to me.

Answer 3

I was just trying the same and with the help of these two attributes setMaxNumberOfMessages and setWaitTimeSeconds i was able to get 10 messages.

ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
                      receiveMessageRequest.setMaxNumberOfMessages(10);
                      receiveMessageRequest.setWaitTimeSeconds(20);

Snapshot of o/p:

Receiving messages from TestQueue.
Number of messages:10
Message
MessageId:     31a7c669-1f0c-4bf1-b18b-c7fa31f4e82d 
...

Answer 4

Here's a workaround, you can call receiveMessageFromSQS method asynchronously.

   bulkReceiveFromSQS (queueUrl, totalMessages, asyncLimit, batchSize, visibilityTimeout, waitTime, callback) {
    batchSize = Math.min(batchSize, 10);

    let self = this,
        noOfIterations = Math.ceil(totalMessages / batchSize);

    async.timesLimit(noOfIterations, asyncLimit, function(n, next) {
        self.receiveMessageFromSQS(queueUrl, batchSize, visibilityTimeout, waitTime,
            function(err, result) {
                if (err) {
                    return next(err);
                }

                return next(null, _.get(result, 'Messages'));
            });
    }, function (err, listOfMessages) {
        if (err) {
            return callback(err);
        }
        listOfMessages = _.flatten(listOfMessages).filter(Boolean);

        return callback(null, listOfMessages);
    });
}

It will return you an array with a given number of messages

Answer 5

AWS API Reference Guide: Query/QueryReceiveMessage

Due to the distributed nature of the queue, a weighted random set of machines is sampled on a ReceiveMessage call. That means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.

and

MaxNumberOfMessages: Maximum number of messages to return. SQS never returns more messages than this value but might return fewer.

Answer 6

receiveMessageRequest.withMaxNumberOfMessages(10);

Just to be clear, the more practical use of this would be to add to your constructor like this:

ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl).withMaxNumberOfMessages(10);

Otherwise, you might as well just do:

receiveMessageRequest.setMaxNumberOfMessages(10);

That being said, changing this won't help the original problem.