How to use Task.WhenAny and implement retry

Question

I have a solution that creates multiple I/O based tasks and I\'m using Task.WhenAny() to manage these tasks. But often many of the tasks will fail due to network issue or reques

Answer 1

I believe it would be easier to retry within the tasks, and then replace the Task.WhenAny-in-a-loop antipattern with Task.WhenAll

E.g., using Polly:

var tasks = new List<Task<MyType>>();
var policy = ...; // See Polly documentation
foreach(var item in someCollection)
  tasks.Add(policy.ExecuteAsync(() => GetSomethingAsync()));
await Task.WhenAll(tasks);

or, more succinctly:

var policy = ...; // See Polly documentation
var tasks = someCollection.Select(item => policy.ExecuteAsync(() => GetSomethingAsync()));
await Task.WhenAll(tasks);

Answer 2

If you don't want to use the Polly library for some reason, you could use the Retry method bellow. It accepts a task factory, and keeps creating and then awaiting a task until it completes successfully, or the maxAttempts have been reached:

public static async Task<TResult> Retry<TResult>(Func<Task<TResult>> taskFactory,
    int maxAttempts)
{
    int failedAttempts = 0;
    while (true)
    {
        try
        {
            var task = taskFactory();
            return await task.ConfigureAwait(false);
        }
        catch
        {
            failedAttempts++;
            if (failedAttempts >= maxAttempts) throw;
        }
    }
}

You could then use this method to download (for example) some web pages.

string[] urls =
{
    "https://stackoverflow.com",
    "https://superuser.com",
    //"https://no-such.url",
};
var httpClient = new HttpClient();
var tasks = urls.Select(url => Retry(async () =>
{
    return (Url: url, Html: await httpClient.GetStringAsync(url));
}, maxAttempts: 5));

var results = await Task.WhenAll(tasks);
foreach (var result in results)
{
    Console.WriteLine($"Url: {result.Url}, {result.Html.Length:#,0} chars");
}

Output:

Url: https://stackoverflow.com, 112,276 chars
Url: https://superuser.com, 122,784 chars

If you uncomment the third url then instead of these results an HttpRequestException will be thrown, after five failed attempts.

The Task.WhenAll method will wait for the completion of all tasks before propagating the error. In case it is preferable to report the error as soon as possible, you can find solutions in this question.