How to “keep-alive” with cookielib and httplib in python?

Question

In python, I'm using httplib because it "keep-alive" the http connection (as oppose to urllib(2)). Now, I want to use cookielib with httplib but they seem to hate each other!! (no way to interface them together).

Does anyone know of a solution to that problem?

Answer 1

HTTP handler for urllib2 that supports keep-alive

Answer 2

You should consider using the Requests library instead at the earliest chance you have to refactor your code. In the mean time;

HACK ALERT! :)

I'd go other suggested way, but I've done a hack (done for different reasons though), which does create an interface between httplib and cookielib.

What I did was creating a fake HTTPRequest with minimal required set of methods, so that CookieJar would recognize it and process cookies as needed. I've used that fake request object, setting all the data needed for cookielib.

Here is the code of the class:

class HTTPRequest( object ):
"""
Data container for HTTP request (used for cookie processing).
"""

    def __init__( self, host, url, headers={}, secure=False ):
        self._host = host
        self._url = url
        self._secure = secure
        self._headers = {}
        for key, value in headers.items():
            self.add_header(key, value)

    def has_header( self, name ):
        return name in self._headers

    def add_header( self, key, val ):
        self._headers[key.capitalize()] = val

    def add_unredirected_header(self, key, val):
        self._headers[key.capitalize()] = val

    def is_unverifiable( self ):
        return True

    def get_type( self ):
        return 'https' if self._secure else 'http'

    def get_full_url( self ):
        port_str = ""
        port = str(self._host[1])
        if self._secure:
            if port != 443:
                port_str = ":"+port
        else:
            if port != 80:
                port_str = ":"+port
        return self.get_type() + '://' + self._host[0] + port_str + self._url

    def get_header( self, header_name, default=None ):
        return self._headers.get( header_name, default )

    def get_host( self ):
        return self._host[0]

    get_origin_req_host = get_host

    def get_headers( self ):
        return self._headers

Please note, the class has support for HTTPS protocol only (all I needed at the moment).

The code, which used this class was (please note another hack to make response compatible with cookielib):

cookies = CookieJar()

headers = {
    # headers that you wish to set
}

# construct fake request
fake_request = HTTPRequest( host, request_url, headers )

# add cookies to fake request
cookies.add_cookie_header(fake_request)

# issue an httplib.HTTPConnection based request using cookies and headers from the fake request
http_connection.request(type, request_url, body, fake_request.get_headers())

response = http_connection.getresponse()

if response.status == httplib.OK:
    # HACK: pretend we're urllib2 response
    response.info = lambda : response.msg

    # read and store cookies from response
    cookies.extract_cookies(response, fake_request)

    # process response...