Unable to download NSE data using python

Question

I am trying to download the NSE equities data("https://www.nseindia.com/market-data/live-equity-market?symbol=NIFTY%2050")

When I paste the below URL in the browser the file gets downloaded. https://www.nseindia.com/api/equity-stockIndices?csv=true&index=SECURITIES%20IN%20F%26O

When I'm trying to download the same file using python's requests package it's going in a forever loop.

Here is the code I used to download the file:

def download_url(url, save_path, chunk_size=1024):
    """
    r = requests.get(url, stream=True, verify=False)
    if r.status_code == 200:
        with open(save_path, 'wb') as fd:
            for chunk in r.iter_content(chunk_size=chunk_size):
                fd.write(chunk)
    """
    try:
        r = requests.get(url, stream=True, verify=False)
    except HTTPError as http_err:
        print(f'Adjacent Error occurred while accessing URL:{http_err}')
    except Exception as err:
        print(f'Adjacent error occurred while accessing URL:{err}')
    else:
        with open(save_path, 'wb') as fd:
            for chunk in r.iter_content(chunk_size=chunk_size):
                fd.write(chunk)

Answer 1

You need to have loaded the page's cookies in order to execute the request. You can do that by vising the main page first, then trying to load the API request:

from requests import Session

# Session to hold cookies
s = Session()
# Emulate browser
s.headers.update({"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"})

# Get the cookies from the main page (will update automatically in headers)
s.get("https://www.nseindia.com/")

# Get the API data
data = s.get("https://www.nseindia.com/api/equity-stockIndices?csv=true&index=SECURITIES IN F%26O").text

# Write to file
with open("securities.csv", "w") as f:
    f.write(data)