Logging in and using cookies in pycurl

Question

I need to download a file that is on a password protected page. To get to the page manually I first have to authenticate via an ordinary login page. I want to use curl to fetc

Answer 1

wds is right on.

for your further edification, the available options are based on those at http://curl.haxx.se/libcurl/c/curl_easy_setopt.html (see the section on cookie shortcuts).

a 500 is an internal server error...hard to be sure whether this can be blamed on your script without knowing more information about what's going on here. you could be failing to pass other data the page is expecting (unrelated to cookies) for all we know (and they have not implemented graceful error handling!)

jb

Answer 2

You should store cookie first and then read from it:

C.setopt(pycurl.COOKIEJAR, 'cookie.txt')
C.setopt(pycurl.COOKIEFILE, 'cookie.txt')

Here what curl --help returned:

-b, --cookie STRING/FILE  String or file to read cookies from (H)
-c, --cookie-jar FILE  Write cookies to this file after operation (H)

See this sample:

def connect(self):
    '''
    Connect to NGNMS server
    '''
    host_url = self.ngnms_host + '/login'

    c = pycurl.Curl()
    c.setopt(c.URL, host_url)
    c.setopt(pycurl.TIMEOUT, 10)

    c.setopt(pycurl.FOLLOWLOCATION, 1)
    c.setopt(pycurl.POSTFIELDS, 'j_username={ngnms_user}&j_password={ngnms_password}'.format(**self.ngnms_login))
    c.setopt(pycurl.COOKIEJAR, 'data/ngnms.cookie')

    # c.setopt(c.VERBOSE, True)

    c.setopt(pycurl.SSL_VERIFYPEER, 0);
    session = c
    return session

Answer 3

I believe Curl will store the cookies but you need to use them explicitly. I've only ever used the command line interface for this though. Scanning the documentation I think you might want to try:

C.setopt(pycurl.COOKIEFILE, 'cookie.txt')

(before the second request)