Logging in and using cookies in pycurl

Question

I need to download a file that is on a password protected page. To get to the page manually I first have to authenticate via an ordinary login page. I want to use curl to fetch this page in script.
My script first logins. It appears to succeed--it returns a 200 from a PUT to /login. However, the fetch of the desired page fails, with a 500.

I am using a "cookie jar":

C.setopt(pycurl.COOKIEJAR, 'cookie.txt')

In verbose mode, I can see cookies being exchanged when I fetch the file I need. Now my question: Is there more to using a COOKIEJAR?

Answer 1

I believe Curl will store the cookies but you need to use them explicitly. I've only ever used the command line interface for this though. Scanning the documentation I think you might want to try:

C.setopt(pycurl.COOKIEFILE, 'cookie.txt')

(before the second request)

Answer 2

You should store cookie first and then read from it:

C.setopt(pycurl.COOKIEJAR, 'cookie.txt')
C.setopt(pycurl.COOKIEFILE, 'cookie.txt')

Here what curl --help returned:

-b, --cookie STRING/FILE  String or file to read cookies from (H)
-c, --cookie-jar FILE  Write cookies to this file after operation (H)

See this sample:

def connect(self):
    '''
    Connect to NGNMS server
    '''
    host_url = self.ngnms_host + '/login'

    c = pycurl.Curl()
    c.setopt(c.URL, host_url)
    c.setopt(pycurl.TIMEOUT, 10)

    c.setopt(pycurl.FOLLOWLOCATION, 1)
    c.setopt(pycurl.POSTFIELDS, 'j_username={ngnms_user}&j_password={ngnms_password}'.format(**self.ngnms_login))
    c.setopt(pycurl.COOKIEJAR, 'data/ngnms.cookie')

    # c.setopt(c.VERBOSE, True)

    c.setopt(pycurl.SSL_VERIFYPEER, 0);
    session = c
    return session

Answer 3

wds is right on.

for your further edification, the available options are based on those at http://curl.haxx.se/libcurl/c/curl_easy_setopt.html (see the section on cookie shortcuts).

a 500 is an internal server error...hard to be sure whether this can be blamed on your script without knowing more information about what's going on here. you could be failing to pass other data the page is expecting (unrelated to cookies) for all we know (and they have not implemented graceful error handling!)

jb