HttpClient async requests not completing for large batch sent out in a loop

Question

I think I\'ve managed to make a test that shows this problem repeatably, at least on my system. This question relates to HttpClient being used for a bad endpoint (no

Answer 1

Your exception information is being lost in the WhenAll task. Instead of using that, try this:

Task aggregateTask = Task.Factory.ContinueWhenAll(
    Batch,
    TaskExtrasExtensions.PropagateExceptions,
    TaskContinuationOptions.ExecuteSynchronously);

aggregateTask.Wait();

This uses the PropagateExceptions extension method from the Parallel Extensions Extras sample code to ensure that exception information from the tasks in the batch operation are not lost:

/// <summary>Propagates any exceptions that occurred on the specified tasks.</summary>
/// <param name="tasks">The Task instances whose exceptions are to be propagated.</param>
public static void PropagateExceptions(this Task [] tasks)
{
    if (tasks == null) throw new ArgumentNullException("tasks");
    if (tasks.Any(t => t == null)) throw new ArgumentException("tasks");
    if (tasks.Any(t => !t.IsCompleted)) throw new InvalidOperationException("A task has not completed.");
    Task.WaitAll(tasks);
}

Answer 2

I looked through the source of HttpClient using reflector. For the synchronously executed part of the operation (when it is kicked-off), there seems to be no timeout applied to the returned task, as far as I can see. There is some timeout implementation that calls Abort() on an HttpWebRequest object, but again they seem to have missed out any timeout cancellation or faulting of the returned task on this side of the async function. There maybe something on the callback side, but sometimes the callback is probably "going missing", leading to the returned Task never completing.

I posted a question asking how to add a timeout to any Task, and an answerer gave this very nice solution (here as an extension method):

public static Task<T> WithTimeout<T>(this Task<T> task, TimeSpan timeout)
{
    var delay = task.ContinueWith(t => t.Result
        , new CancellationTokenSource(timeout).Token);
    return Task.WhenAny(task, delay).Unwrap();
}

So, calling HttpClient like this should prevent any "Tasks gone bad" from never ending:

Task<HttpResponseMessage> _response = httpClient.PostAsJsonAsync<Object>(Target, new Object()).WithTimeout<HttpResponseMessage>(httpClient.Timeout);

A couple more things that I think made requests less likely to go missing: 1. Increasing the timeout from 3s to 30s made all the tasks finish in the program that I posted with this question. 2. Increasing the number of concurrent connections allowed using for example System.Net.ServicePointManager.DefaultConnectionLimit = 100;

Answer 3

I came across this question when googling for solutions to a similar problem from WCF. That series of exceptions is exactly the same pattern I see. Eventually through a ton of investigation I found a bug in HttpWebRequest that HttpClient uses. The HttpWebRequest gets in a bad state and only sends the HTTP headers. It then sits waiting for a response which will never be sent.

I've raised a ticket with Microsoft Connect which can be found here: https://connect.microsoft.com/VisualStudio/feedback/details/1805955/async-post-httpwebrequest-hangs-when-a-socketexception-occurs-during-setsocketoption

The specifics are in the ticket but it requires an async POST call from the HttpWebRequest to a non-localhost machine. I've reproduced it on Windows 7 in .Net 4.5 and 4.6. The failed SetSocketOption call, which raises the SocketException, only fails on Windows 7 in testing.

For us the UseNagleAlgorithm setting causes the SetSocketOption call, but we can't avoid it as WCF turns off UseNagleAlgorithm and you can't stop it. In WCF it appears as a timed out call. Obviously this isn't great as we're spending 60s waiting for nothing.