What's the proper way to abandon an Azure SB Message so that it becomes visible again in the future in a way I can control?

Question

So the scenario is that I'm using an SB queue to throttle outgoing callbacks to other services. One of the standard problems with calling back to other services is that they may be down for uncontrollable amounts of time. Assuming I detect that the target is down/not responding, what is the best pattern for abandoning that message so that it doesn't reappear on the queue immediately?

Here's are some approaches I'm either aware of, have tried or am considering:

• Obviously if I just use BrokeredMessage::Abandon() the message will be unlocked and put back on the queue. This is obviously undesirable for this scenario and what I'm trying to avoid.

• If I just ignore the fact that I ran into an error and never call Abandon this will keep it from showing up immediately, but I don't really have fine grained control over how long until it shows up again and I would like to implement a decaying retry strategy.

• I thought maybe I could call BrokeredMessage::Abandon(IDictionary<string, object>) and somehow update the ScheduledEnqueueTimeUTC property, but I have tried this and there doesn't seem to be a way to affect that property beyond the initial sending of the message. Makes sense, but thought worth a try.

• I have considered just using BrokeredMessage::Complete() in this situation and actually just enqueueing a new copy of the message with the ScheduledEqueueTimeUTC property set.

The final bullet almost seems too heavy handed, but I'm coming to the conclusion it's probably the right answer given the inherent nature of queues. I just figured there might be a nicer way to do this within Azure SB queues that I'm missing.

Answer 1

If you want to put a message away for a while and you have a place to write down the SequenceNumber (which might be in Session state in a sessionful queue), you can Defer() it. Deferred messages can be retrieved using a special Receive overload giving the SequenceNumber; that's also the only way to get at them again other than them expiring, so careful. This feature was built for workflows and state machines so that they can deal with out of order message arrival.

Answer 2

In bullet #2 above: Can't you just set the TTL timespan on the message when calling Receive(TimeSpan) instead of the default Receive()? Then, you can simply abandon the message (without calling Abandon()), and the message should reappear in the queue when the TTL expires. While this doesn't give you fine-grain control to say "Reappear after x seconds," it does give you predictability for when the message does reappear.

Note: With Storage-based queues, you can update the invisibility timeout, so that would give you fine-grain control for message re-appearance.

Answer 3

Feels a bit hacky, but the solution I came up with is

try 
{
   ...
}
catch (Exception ex)
{
   await Task.Delay(30000);
   throw;
}

This way it will wait for 30 seconds before allowing it to abandon. It will eventually dead letter after the configured amount of times.

I am using Azure Webjobs for receiving. Although I am using Task.Delay instead of Thread.Sleep it doesn't seem to be freeing up the thread to process another item from the queue while it awaits (by default, Webjobs processes 16 in parallel).

Answer 4

If I were you I would consult one of the many Enterprise integration patterns pages that are around on the internet for a solution. Basically you want to have a retry which if it fails successively sends the message to a dead letter queue. These messages can then we requeued at a later date. This could be manual or automated depending on requirements.

Please note that while the page I have sent you is related to camel and therefore java everything described on that page is applicable .NET and azure. Here is a more .NET one if your interested http://www.eaipatterns.com/

Answer 5

I would prefer the last approach because it seems to be the most simple solution using the built in features of the Azure service bus.

The flow is this:

var newMessage = new BrokeredMessage();
// Copy message body and properties from original message...

var scheduleTimeUtc = DateTimeOffset.UtcNow.Add(...);
await queueClient.ScheduleMessageAsync(newMessage, scheduleTimeUtc);

await originalMessage.CompleteAsync()