使用 requests 模块模拟 HTTP 发布请求不起作用

Question

I tried to retrieve a page after submitting the form in the following link (with default options):https://selfservice.pasadena.edu/prod/pw_psearch_sched.p_search

I realized that the form is post method, so I used requests module to simulate a post request.

r = requests.post(URL, data=form_data).text

For the form data, I extracted using chrome developer tool.

form_data = {
    'TERM': '202070',
    'TERM_DESC': 'Fall 2020',
    'sel_subj': 'dummy',
    'sel_day': 'dummy',
    'sel_schd': 'dummy',
    'sel_camp': 'dummy',
    'sel_ism': 'dummy',
    'sel_sess': 'dummy',
    'sel_instr': 'dummy',
    'sel_ptrm': 'dummy',
    'sel_attrib': 'dummy',
    'being_hh': '5',
    'begin_mi': '0',
    'begin_ap': 'a',
    'end_hh': '11',
    'end_mi': '0',
    'end_ap': 'p',
    'aa': 'N',
    'bb': 'N',
    'ee': 'N'
}

But it keeps giving me 404 Error .

At first, I tried to use selenium to simulate clicking the submit button but it is way too slow.

Any help would be appreciated.

Answer 1

There are two problems

1. form sends to different url: https://selfservice.pasadena.edu/prod/pw_psearch_sched.p_listthislist

2. small mistake in name 'begin_hh' - you have 'being_hh'

---

Working code

import requests

data = {
    'TERM': '202070',
    'TERM_DESC': 'Fall 2020',
    'sel_subj':   'dummy',
    'sel_day':    'dummy',
    'sel_schd':   'dummy',
    'sel_camp':   'dummy',
    'sel_ism':    'dummy',
    'sel_sess':   'dummy',
    'sel_instr':  'dummy',
    'sel_ptrm':   'dummy',
    'sel_attrib': 'dummy',

    'sel_crse': '',
    'sel_crn': '',
    'sel_title': '',

    'begin_hh': '5',
    'begin_mi': '0',
    'begin_ap': 'a',
    'end_hh': '11',
    'end_mi': '0',
    'end_ap': 'p',
    'aa': 'N',
    'bb': 'N',
    'ee': 'N',
}

URL = 'https://selfservice.pasadena.edu/prod/pw_psearch_sched.p_listthislist'
r = requests.post(URL, data=data)
print(r.status_code)
#print(r.text)

---

BTW: originally form sends two values for some fields and it would be

'sel_subj':   ['dummy', '%'],
'sel_day':    'dummy',
'sel_schd':   ['dummy', '%'],
'sel_camp':   ['dummy', '%'],
'sel_ism':    ['dummy', '%'],
'sel_sess':   ['dummy', '%'],
'sel_instr':  ['dummy', '%'],
'sel_ptrm':   ['dummy', '%'],
'sel_attrib': ['dummy', '%'],

but page works even if you send one value.

---

BTW: in DevTool you can get code for rpogram curl and later you can convert it to Python code using page https://curl.trillworks.com/