HTTP Download very Big File

Question

I\'m working at a web application in Python/Twisted.

I want the user to be able to download a very big file (> 100 Mb). I don\'t want to load all the file in memory (of

Answer 1

If this really is text/plain content, you should seriously consider sending it with Content-Encoding: gzip whenever a client indicates they can handle it. You ought to see huge bandwidth savings. Additionally, if this is a static file, what you really want to do is use sendfile(2). As for browsers not doing what you expect in terms of downloading things, you might want to look at the Content-Disposition header. So anyhow, the logic goes like this:

If the client indicates they can handle gzip encoding via the Accept-Encoding header (e.g. Accept-Encoding: compress;q=0.5, gzip;q=1.0 or Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0 or similar) then compress the file, cache the compressed result somewhere, write the correct headers for the response (Content-Encoding: gzip, Content-Length: n, Content-Type: text/plain, etc), and then use sendfile(2) (however that may or may not have been made available in your environment) to copy the content from the open file descriptor into your response stream.

If they don't accept gzip, do the same thing, but without gzipping first.

Alternatively, if you have Apache, Lighttpd, or similar acting as a transparent proxy in front of your server, you could use the X-Sendfile header, which is exceedingly fast:

response.setHeader('Content-Type', 'text/plain')
response.setHeader(
  'Content-Disposition',
  'attachment; filename="' + os.path.basename(fileName) + '"'
)
response.setHeader('X-Sendfile', fileName)
response.setHeader('Content-Length', os.stat(fileName).st_size)