how to time-out gracefully while downloading with python

Question

I\'m downloading a huge set of files with following code in a loop:

try:
    urllib.urlretrieve(url2download, destination_on_local_filesystem)
except KeyboardInt         


        

Answer 1

Try:

import socket

socket.setdefaulttimeout(30)

Answer 2

There's a discussion of this here. Caveats (in addition to the ones they mention): I haven't tried it, and they're using urllib2, not urllib (would that be a problem for you?) (Actually, now that I think about it, this technique would probably work for urllib, too).

Answer 3

This question is more general about timing out a function: How to limit execution time of a function call in Python

I've used the method described in my answer there to write a wait for text function that times out to attempt an auto-login. If you'd like similar functionality you can reference the code here:

http://code.google.com/p/psftplib/source/browse/trunk/psftplib.py

Answer 4

If you're not limited to what's shipped with python out of the box, then the urlgrabber module might come in handy:

import urlgrabber
urlgrabber.urlgrab(url2download, destination_on_local_filesystem,
                   timeout=30.0)