How to download a file with urllib3?

Question

This is based on another question on this site: What's the best way to download file using urllib3 However, I cannot comment there so I ask another question:

How to

Answer 1

You were very close, the piece that was missing is setting preload_content=False (this will be the default in an upcoming version). Also you can treat the response as a file-like object, rather than the .data attribute (which is a magic property that will hopefully be deprecated someday).

- with http.request('GET', url) ...
+ with http.request('GET', url, preload_content=False) ...

This code should work:

http = urllib3.PoolManager()

with http.request('GET', url, preload_content=False) as r, open(path, 'wb') as out_file:       
    shutil.copyfileobj(r, out_file)

urllib3's response object also respects the io interface, so you can also do things like...

import io
response = http.request(..., preload_content=False)
buffered_response = io.BufferedReader(response, 2048)

As long as you add preload_content=False to any of your three attempts and treat the response as a file-like object, they should all work.

It is unfortunate that the urllib documentation does not cover the best practice in this topic.

You're totally right, I hope you'll consider helping us document this use case by sending a pull request here: https://github.com/shazow/urllib3