Python请求模块未在会话中传递参数

Question

I am using am attempting to do a bulk download of a series of PDFs from a site that requires login authentication. I am able to successfully log in, however, when I attempt a GET request for '/transcripts/transcript.pdf?user_id=3007' but, the request returns the content for '/transcripts/transcript.pdf' .

Does anyone have any idea why the URL param is not sending? Or why it would be rerouted?

I have tried passing the parameter 'user_id' as data, params, and hardcoded in the URL.

I have removed the actual domain from the strings below just for privacy

with requests.Session() as s:
    login = s.get('<domain>/login/canvas')
    # print the html returned or something more intelligent to see if it's a successful login page.
    print(login.text)
    login_html = lxml.html.fromstring(login.text)
    hidden_inputs = login_html.xpath(r'//form//input[@type="hidden"]')
    form = {x.attrib["name"]: x.attrib["value"] for x in hidden_inputs}
    print("form: ",form)
    form['pseudonym_session[unique_id]']= username 
    form['pseudonym_session[password]']= password
    response = s.post('<domain>/login/canvas',data=form)
    print(response.url, response.status_code) # gets <domain>?login_success=1 200


    # An authorised request.
    data = { 'user_id':'3007'}
    r = s.get('<domain>/transcripts/transcript.pdf?user_id=3007', data=data)
    print(r.url) # gets <domain>/transcripts/transcript.pdf
    print(r.status_code) # gets 200
    with open('test.pdf', 'wb') as f:
        f.write(r.content)

GET response returns /transcripts/transcript.pdf and not /transcripts/transcript.pdf?user_id=3007

Answer 1

From the looks of it, you are trying to use canvas. I'm pretty sure in canvas, you can bulk download all test attachments.

If that's not the case, There are a few things to try:

1. after logging in, try typing the url with user_id into a browser. Does that take you directly to the PDF file or links to one?
2. if so, look at the url, it may simply not display the parameters; some websites do this, don't worry about it

If not, GET may not be enough; perhaps the site uses javascript, etc.

Answer 2

after looking through the '.history' of the request I found a series of 302 redirects.
The first was to '/login?force_login=0&target_uri=%2Ftranscripts%2Ftranscript.pdf'
In a desperate attempt, I tried:
s.get('/login?force_login=0&target_uri=%2Ftranscripts%2Ftranscript.pdf%3Fuser_id%3D3007') and this still rerouted me a few times but ultimately got me the file I wanted!

If anyone has a more elegant solution to this or any resources that I can read I would greatly appreciate it!