Python获取请求并从搜索中检索数据

Question

I'm trying to use the requests module to retrieve data from this website: https://toelatingen.ctgb.nl/

I want to receive the found data when I put in "11462" the "Zoekterm" field for example.

data = { "searchTerm": "11462"}
session = requests.Session()
r = session.post('https://toelatingen.ctgb.nl/',data=data)

body_data = r.text

The content of the body_data does not, unfortunately, contain the information searched for.

Thanks for helping me.

Answer 1

The reason you're not getting the response data is because the site doesn't do the search at that url. Instead it makes a call to https://toelatingen.ctgb.nl/nl/admissions/overview .

When you're trying to get information off the internet the first thing you want to do is check how your web browser is getting the data. If you open up whatever inspection tool comes with your browser of choice (typically the hotkey is ctrl+shift+i), you should be able to find a Network tab that tracks the requests and responses the browser makes. Once you have that open, get your browser to display the information you want and watch the Network Tab while it's doing so. Check whatever responses come up to find the one that has the information you want and then replicate the request your browser used.

In your case:

• The root page loads an empty page first from https://toelatingen.ctgb.nl/
• It then loads a bunch of static files (mostly woff and js; these are used for styling the webpage and handling different proceedures)
• Then it makes a call to https://toelatingen.ctgb.nl/nl/admissions/overview . We can be pretty sure this is the call we want at this point because the response is a json which contains the information that we see displayed on the screen.
• We then copy out all the information- headers and forms, line for line- from that request, plug it in, and see if the requests module returns the same json.
• If it doesn't then that most likely means we're missing something (most often a CSRF Token or a special Accept-Encoding) and we need to do some more tinkering.
• I would also recommend taking a little bit of time to prune out parts of the request data/headers: most of the time they contain extra terms that the server doesn't actually need. This will save space and give you a better idea of what parts of the request you can change.