Print all incoming XHR requests python

Question

I would like to get all the XHR requests from the URL.

When I inspect the site and go yo Network -> XHR I see multiple XHR links, I would like to get them all once loading has finished. I guess I need a combination of selenium and requests.

I found the following code on SO here but it doesn't give me any output and hands me an error.

I looked for other questions but it doesn't seem like a lot of people had this problem.

Error:

File "C:\ProgramData\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value

My code:

####   Gets all XHRs ####
import requests

url= "https://forsikringsguiden.dk/#!/"

headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36",
           "X-Requested-With" : "XMLHttpRequest",
           "Host" : "forsikringsguiden.dk",
           "Referer" : "https://forsikringsguiden.dk/"}

response = requests.get(url, headers=headers)

xhr = response.json()
print(xhr)

Edit: Do I need to add any parameters to my code?

Answer 1

You'll need to coordinate with the site maintainers. They'll be able to say what headers or parameters you need to send to get a JSON response. Or if they even support JSON at all.

Your original code works given a URL that actually returns JSON:

import requests

url= "http://api.plos.org/search?q=title:DNA"

headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36",
           "X-Requested-With" : "XMLHttpRequest"}

response = requests.get(url, headers=headers)

xhr = response.json()
print(xhr)