Library or tool to download multiple files in parallel [closed]

Answer 1

Nowadays there are excellent Python libs you might want to use - urllib3 and requests

Answer 2

You can try pycurl, though the interface is not easy at first, but once you look at examples, its not hard to understand. I have used it to fetch 1000s of web pages in parallel on meagre linux box.

1. You don't have to deal with threads, so it terminates gracefully, and there are no processes left behind
2. It provides options for timeout, and http status handling.
3. It works on both linux and windows.

The only problem is that it provides a basic infrastructure (basically just a python layer above the excellent curl library). You will have to write few lines to achieve the features as you want.

Answer 3

This seems pretty flexible:

http://keramida.wordpress.com/2010/01/19/parallel-downloads-with-python-and-gnu-wget/

Answer 4

Threading isn't "half-assed" unless you're a bad programmer. The best general approach to this problem is the producer / consumer model. You have one dedicated URL producer, and N dedicated download threads (or even processes if you use the multiprocessing model).

As for all of your requirements, ALL of them CAN be done with the normal python threaded model (yes, even catching Ctrl+C -- I've done it).