C# Stream Response from 3rd party, minimal buffering

Question

Our ASP.NET MVC endpoint is a behaving as a proxy to another 3rd party HTTP endpoint, which returns about 400MB of XML document generated dynamically.

Is there a way

Answer 1

You can reduce the memory footprint by not buffering and just copying the stream directly to output stream, an quick n' dirty example of this here:

    public async Task<ActionResult> Download()
    {
        using (var httpClient = new System.Net.Http.HttpClient())
        {
            using (
                var stream = await httpClient.GetStreamAsync(
                    "https://ckannet-storage.commondatastorage.googleapis.com/2012-10-22T184507/aft4.tsv.gz"
                    ))
            {
                Response.ContentType = "application/octet-stream";
                Response.Buffer = false;
                Response.BufferOutput = false;
                await stream.CopyToAsync(Response.OutputStream);
            }
            return new HttpStatusCodeResult(200);
        }
    }

If you want to reduce the footprint even more you can set a lower buffer size with the CopyToAsync(Stream, Int32) overload, default is 81920 bytes.

Answer 2

My requirement on proxy download also need to ensure the source ContentType (or any Header you need) can be forwarded as well. (e.g. If I proxy-download a video in http://techslides.com/sample-webm-ogg-and-mp4-video-files-for-html5, I need to let user see the same browser-video-player screen as they open the link directly, but not jumping to file-download / hard-coded ContentType)

Basing on answer by @devlead + another post https://stackoverflow.com/a/30164356/4684232, I adjusted a lil on the answer to fulfill my need. Here's my adjusted code in case anyone has the same need.

public async Task<ActionResult> Download(string url)
{
    using (var httpClient = new System.Net.Http.HttpClient())
    {
        using (var response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
        {
            response.EnsureSuccessStatusCode();
            using (var stream = await response.Content.ReadAsStreamAsync())
            {
                Response.ContentType = response.Content.Headers.ContentType.ToString();
                Response.Buffer = false;
                Response.BufferOutput = false;
                await stream.CopyToAsync(Response.OutputStream);
            }
        }

        return new HttpStatusCodeResult(200);
    }
}

p.s. HttpCompletionOption.ResponseHeadersRead is the important performance key. Without it, GetAsync will await until the whole source response stream are downloaded, which is much slower.