Library or tool to download multiple files in parallel [closed]

Answer 1

Try using aria2 through simple python subprocess module. It provide all requirements from your list, except 7, out of the box, and 7 is easy to write. aria2c has a nice xml-rpc or json-rpc interface to interact with it from your scripts.

Answer 2

I used the standard libs for that, urllib.urlretrieve to be precise. downloaded podcasts this way, via a simple thread pool, each using its own retrieve. I did about 10 simultanous connections, more should not be a problem. Continue a interrupted download, maybe not. Ctrl-C could be handled, I guess. Worked on Windows, installed a handler for progress bars. All in all 2 screens of code, 2 screens for generating the URLs to retrieve.

Answer 3

There are lots of options but it will be hard to find one which fits all your needs.

In your case, try this approach:

1. Create a queue.
2. Put URLs to download into this queue (or "config objects" which contain the URL and other data like the user name, the destination file, etc).
3. Create a pool of threads
4. Each thread should try to fetch a URL (or a config object) from the queue and process it.

Use another thread to collect the results (i.e. another queue). When the number of result objects == number of puts in the first queue, then you're finished.

Make sure that all communication goes via the queue or the "config object". Avoid accessing data structures which are shared between threads. This should save you 99% of the problems.

Answer 4

I would suggest Twisted, although it is not a ready made solution, but provides the main building blocks to get every feature you listed in an easy way and it does not use threads.

If you are interested, take a look at the following links:

• http://twistedmatrix.com/documents/current/api/twisted.web.client.html#getPage
• http://twistedmatrix.com/documents/current/api/twisted.web.client.html#downloadPage

As per your requirements:

1. Supported out of the box
2. Supported out of the box
3. Supported out of the box
4. Timeout supported out of the box, other error handling done through deferreds
5. Achieved easily using cooperators (example 7)
6. Supported out of the box
7. Not supported, solutions exists (and they are not that hard to implement)
8. Not supported, it can be implemented (but it will be relatively hard)

Answer 5

I don't think such a complete library exists, so you'll probably have to write your own. I suggest taking a look at gevent for this task. They even provide a concurrent_download.py example script. Then you can use urllib2 for most of the other requirements, such as handling HTTP status codes, and displaying download progress.

Answer 6

Does urlgrabber fit your requirements?

http://urlgrabber.baseurl.org/

If it doesn't, you could consider volunteering to help finish it. Contact the authors, Michael Stenner and Ryan Tomayko.

Update: Googling for "parallel wget" yields these, among others:

http://puf.sourceforge.net/

http://www.commandlinefu.com/commands/view/3269/parallel-file-downloading-with-wget

It seems like you have a number of options to choose from.