Python requests GET not getting the JSON payload?

Question

I am trying to get the JSON data from the following URL:

import requests as r

url = "https://www.nseindia.com/json/CorporateFiling/CF-corpactions-equity.json"

header = {
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0',
        "X-Requested-With": "XMLHttpRequest"
    }
resp = r.get(url, stream=True, timeout=20, headers=header)
j = resp.json()

I get the JSON from doing this, but in the inspection I see the data is in the Response payload part, which is not in j .

I have never faced this problem before and my search lead me to POST questions.

Answer 1

I tested it using postman

User-Agent value is your problem

you could simply remove it and it will work

Answer 2

I might be wrong and didn't get question correctly, but compering data getting from UI and compering data getting from API are the same:

import json
import requests
from selenium import webdriver

url = 'https://www.nseindia.com/json/CorporateFiling/CF-corpactions-equity.json'
header = {
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0',
        "X-Requested-With": "XMLHttpRequest"
    }
driver = webdriver.Chrome()
driver.get(url)
content = driver.find_element_by_xpath('//pre').text
driver.quit()
response = requests.get(url,
                        stream=True,
                        timeout=20,
                        headers=header
                        )
print(json.loads(content) == response.json())
assert json.loads(content) == response.json()