python-requests无法获取JSESSIONID和SessionData cookie

Question

I want to scrape a pdf file from http://www.jstor.org/stable/pdf/10.1086/512825.pdf but it wants me to accept Terms and Conditions. While downloading from browser I found out that JSTOR saves my acceptance in 2 cookies with names JSESSIONID and SessionData but python-requests does not grab these two cookie( It grab two other cookies but not these).

Here is my session instantiation code:

def get_raw_session():
    session = requests.Session()
    session.headers.update({'User-Agent': UserAgent().random})
    session.headers.update({'Connection': 'keep-alive'})
    return session

Note that I used python-requests for login-required sites several times before and it worked great but in this case it's not.

I guess problem is that JSTOR is built with jsp and python-requests does not support that.

Any Idea?

Answer 1

The following code is working perfectly fine for me -

import requests
from bs4 import BeautifulSoup

s = requests.session()
r = s.get('http://www.jstor.org/stable/pdf/10.1086/512825.pdf')
soup = BeautifulSoup(r.content)
pdfurl = 'http://www.jstor.org' + soup.find('a', id='acptTC')['href']
with open('export.pdf', 'wb') as handle:
    response = s.get(pdfurl, stream=True)
    for block in response.iter_content(1024):
        if not block:
            break
        handle.write(block)