In-order message processing on serverless Azure Functions using Logic Apps

问题

I need to process incoming messages on Azure. Each message will be associated with a specific entity – say, through an EntityId property – and messages belonging to the same entity must be processed in-order with respect to each other. At the same time, I would to preserve the serverless aspect of Azure Functions; if I have steady streams of messages for 1,000 entities, I'd like to have 1,000 concurrent executions of my function. I haven't found a clean way of achieving this. Service Bus queues have sessions, which are the closest implementation of my requirements, but they are not supported in Azure Functions: https://github.com/Azure/azure-functions-host/issues/563. However, they seem to be supported in Azure Logic Apps. I'm thinking of creating an Azure Logic App that is triggered by a Service Bus queue using sessions ("Correlated in-order delivery using service bus sessions" template), then hooks to an HTTP-triggered Azure Function for processing messages. The Logic App's only purpose is to prevent multiple messages belonging to the same entity/session from being processed concurrently. Can someone provide some insight on whether this approach would work, and whether it has any caveats?

回答1:

Have a look at this article from one of the Azure Function Team Member: In order event processing with Azure Functions

It uses Azure Functions and Azure Events Hubs:

Azure Events Hubs can handle billions of events, and has guarantees around consistency and ordering per partition.

For your scenario, each message related to the same EntityId has to go to the same partition.

The trick to have in order processing and let azure function scale independently is by pulling batches from event hubs, and preserve order.

You function should looks like this:

[FunctionName("EventHubTrigger")]
public static async Task RunAsync([EventHubTrigger("ordered", Connection = "EventHub")] EventData[] eventDataSet, TraceWriter log)
{
    log.Info($"Triggered batch of size {eventDataSet.Length}");
    foreach (var eventData in eventDataSet)
    {
        try
        {
            // Process message in order here.
        }
        catch
        {
            // handle event exception
        }
    }
}

I would recommend you to read the whole article, it is very instructive

You can find the whole solution on Github:

https://github.com/jeffhollan/functions-csharp-eventhub-ordered-processing

来源：https://stackoverflow.com/questions/53572550/in-order-message-processing-on-serverless-azure-functions-using-logic-apps