Close urllib2 connection

Question

I\'m using urllib2 to load files from ftp- and http-servers.

Some of the servers support only one connection per IP. The problem is, that urllib2 does not close the

Answer 1

Biggie: I think it's because the connection is not shutdown().

Note close() releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call shutdown() before close().

You could try something like this before f.close():

import socket
f.fp._sock.fp._sock.shutdown(socket.SHUT_RDWR)

(And yes.. if that works, it's not Right(tm), but you'll know what the problem is.)

Answer 2

The cause is indeed a file descriptor leak. We found also that with jython, the problem is much more obvious than with cpython. A colleague proposed this sollution:

 

    fdurl = urllib2.urlopen(req,timeout=self.timeout)
    realsock = fdurl.fp._sock.fp._sock** # we want to close the "real" socket later 
    req = urllib2.Request(url, header)
    try:
             fdurl = urllib2.urlopen(req,timeout=self.timeout)
    except urllib2.URLError,e:
              print "urlopen exception", e
    realsock.close() 
    fdurl.close()

The fix is ugly, but does the job, no more "too many open connections".

Answer 3

as for Python 2.7.1 urllib2 indeed leaks a file descriptor: https://bugs.pypy.org/issue867

Answer 4

Alex Martelli answers to the similar question. Read this : should I call close() after urllib.urlopen()?

In a nutshell:

import contextlib

with contextlib.closing(urllib.urlopen(u)) as x:
    # ...