Can I directly stream from HttpResponseMessage to file without going through memory?

Question

My program uses HttpClient to send a GET request to a Web API, and this returns a file.

I now use this code (simplified) to store the file to disc:

p         


        

Answer 1

Instead of GetAsync(Uri) use the the GetAsync(Uri, HttpCompletionOption) overload with the HttpCompletionOption.ResponseHeadersRead value.

The same applies to SendAsync and other methods of HttpClient

Sources:

• docs (see remarks) https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.getasync?view=netcore-1.1#System_Net_Http_HttpClient_GetAsync_System_Uri_System_Net_Http_HttpCompletionOption_

The returned Task object will complete based on the completionOption parameter after the part or all of the response (including content) is read.

• .NET Core implementation of GetStreamAsync that uses HttpCompletionOption.ResponseHeadersRead https://github.com/dotnet/corefx/blob/release/1.1.0/src/System.Net.Http/src/System/Net/Http/HttpClient.cs#L163-L168

• HttpClient spike in memory usage with large response

• HttpClient.GetStreamAsync() with custom request? (don't mind the comment on response, the ResponseHeadersRead is what does the trick)

Answer 2

It looks like this is by-design - if you check the documentation for HttpClient.GetAsync() you'll see it says:

The returned task object will complete after the whole response (including content) is read

You can instead use HttpClient.GetStreamAsync() which specifically states:

This method does not buffer the stream.

However you don't then get access to the headers in the response as far as I can see. Since that's presumably a requirement (as you're getting the file name from the headers), then you may want to use HttpWebRequest instead which allows you you to get the response details (headers etc.) without reading the whole response into memory. Something like:

public async Task<bool> DownloadFile()
{
    var uri = new Uri("http://somedomain.com/path");
    var request = WebRequest.CreateHttp(uri);
    var response = await request.GetResponseAsync();

    ContentDispositionHeaderValue contentDisposition;
    var fileName = ContentDispositionHeaderValue.TryParse(response.Headers["Content-Disposition"], out contentDisposition)
        ? contentDisposition.FileName
        : "noname.dat";
    using (var fs = new FileStream(@"C:\test\" + fileName, FileMode.Create, FileAccess.Write, FileShare.None))
    {
        await response.GetResponseStream().CopyToAsync(fs);
    }

    return true
}

Note that if the request returns an unsuccessful response code an exception will be thrown, so you may wish to wrap in a try..catch and return false in this case as in your original example.