Is durable functions suited for high amount of activities?

问题

I have a scenario where I need to calculate 500k activities. All small calculations. Due to throttling I can only compute 30 simulataniously.

Imagine following simple sample:

[FunctionName("Crawl")]
public static async Task<List<string>> RunOrchestrator(
    [OrchestrationTrigger] DurableOrchestrationContext context)
{  
    WriteLine("In orchistration");
    var outputs = new List<string>();

    // here i get 1 million jobs
    var jobs = await context.CallActivityAsync<List<Influencer>>("GetSocialAccountJobs", "");
    var tasks = new Task<string>[jobs.Count];

    var retryOptions = new RetryOptions(
        firstRetryInterval: TimeSpan.FromSeconds(60), maxNumberOfAttempts: 3);

    for (int i = 0; i < jobs.Count; i++)
    {              

        tasks[i] = context.CallActivityWithRetryAsync<string>("Crawl_Hello", retryOptions, jobs[i].Id);
    }

    await Task.WhenAll(tasks);

    return outputs;
}

Everytime an activity is called, this orchestrator function is called. And loops all activities until et finds an activity that has not been called. It will loop millions of millions of times. Am I missing something or is durable functions not suited for this kind of scenarios?

回答1:

Durable Functions auto-scale when running in the Consumption and Elastic Premium plans. There are some documented performance targets that help when deploying durable functions.

Specifically, in your case, you might want to note the following

Unlike fan-out, fan-in operations are limited to a single VM. If your application uses the fan-out, fan-in pattern and you are concerned about fan-in performance, consider sub-dividing the activity function fan-out across multiple sub-orchestrations.

So, instead of calling million activities from a single orchestrator, you could trigger a sub-orchestrator with smaller batches of jobs which in-turn would call the activity function.

来源：https://stackoverflow.com/questions/59249027/is-durable-functions-suited-for-high-amount-of-activities

标签

azure-functions

azure-durable-functions