python 请求：如何通过代理对 post 和访问文件进行身份验证

Question

I am trying to download a research article pdf via a university proxy into which I need to login. I tried following [this answer][1], but the resulting download contains only the login website.

• The article url might look like this: https://iopscience.iop.org/article/10.3847/2041-8213/aaf743/pdf . (this one happens to be open access, but others need to be accessed this way).

• In the browser, I access this through a proxy: https://login.emedien.ub.my-university.edu/login?qurl=https%3a%2f%2fiopscience.iop.org%2farticle%2f10.3847%2f2041-8213%2faaf743%2fpdf . This url is stored in the variable long_proxy in the code sample below.

• In the browser, this brings up a login form:

   <form action="/login" method="post"> <input name="ezproxycsrftoken" type="hidden" value="aBcDeFgH12345"/> <input name="url" type="hidden" value="https://iopscience.iop.org/article/10.3847/2041-8213/aaf743/pdf"> <table> <tr><td>University Username:</td><td><input name="user" style="width:250px" tabindex="1" type="text"/></td></tr> <tr><td>Password:</td><td><input name="pass" style="width:250px" tabindex="2" type="password"/></td></tr> </table> </input> </form>

• Upon entering the username/password, I get forwarded to

  https://iopscience-iop-org.emedien.ub.uni-muenchen.de/article/10.3847/2041-8213/aaf743/pdf

  which brings up the PDF in the browser. I call this url short_proxy in the code sample below.

I try to do that with python requests in the following way:

user_name = 'myname'
passwd = 'mypassword'

with requests.Session() as session:

    session.headers.update({'User-Agent': 'Mozilla/5.0'})

    # Parse the input form for the hidden input
    r2      = requests.get(long_proxy)
    soup    = bs4.BeautifulSoup(r2.text, "html.parser")
    form    = soup.find('form')
    hidden  = form.find('input', attrs={'type':'hidden', 'name':'ezproxycsrftoken'}).attrs['value']
    url_res = form.find('input', attrs={'type':'hidden', 'name':'url'}).attrs['value']

    # set up the login

    payload = {
        'user': user_name,
        'pass': passwd,
        'ezproxycsrftoken': hidden,
        'url': url_res
    }

    # post login

    post = session.post(login, data=payload)

    # get data

    r3 = session.get(short_proxy)
    with open('file.pdf', 'wb') as fid:
        fid.write(r3.content)

• However the downloaded file is not actually a PDF, but turns out to be the html code of the login page.

Any ideas how to get the PDF?

  [1]: https://stackoverflow.com/questions/37816565/python-authentication-with-requests-library-via-post

Answer 1

You're using requests.Session() in order to save cookies/session that the website gives you, yet you're using requests.get() instead of session.get() for your initial request where you fetch your longproxy . Changing your

r2      = requests.get(long_proxy)

to

r2      = session.get(long_proxy)

Should fix your issue. I can not verify this however.

Also note that your long_proxy

https://login.emedien.ub.uni-muenchen.de/login?qurl=https://iopscience.iop.org/article/10.3847/2041-8213/aaf743/

is simply the login url, followed by the pdf url. So you don't really have to fetch that.
This could save you some extra requests / execution time