Python, requests webscraping - NSE India gives emtpy list

Question

I am trying to use requests to scrape data form NSE to find top gainers on Indian stock market. I have used this same method on yahoo finance and it worked but here, I keep getting the empty list as the result.

Here is my code:

import requests
from lxml import html
import urllib3


urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

resp = requests.get('https://www.nseindia.com/market-data/top-gainers-loosers',  verify=False, headers=headers)
tree = html.fromstring(resp.content)

count = 1
stocks = []
for i in range(30):
    name = tree.xpath('//*[@id="topgainer-Table"]/tbody/tr['+str(count)+']/td[1]/a')
    print(name)
    try:
        stocks.append(name[0].text)
    except:
        pass
    #name.text  
    count +=1
    
print(stocks)

As an output I get [ ] printed many times (an empty list). I think that the problem is with NSE because it has many different tables all with same xpaths.

Any ideas?

Answer 1

If you check the website it take some time to load data so from bs4 you will not be able to find data.

Network Tab::

You can find it from Network tab and search for any company name you can get from screenshot attached and find the URL and make call to it and it will return JSON data. so you can extract what so data you want

import requests
import pandas  as pd
headers={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"}
res=requests.get(r"https://www.nseindia.com/api/live-analysis-variations?index=gainers",headers=headers)
all_data=res.json()['NIFTY']['data']
df=pd.DataFrame(all_data)

Output:

URL for top loosers:: https://www.nseindia.com/api/live-analysis-variations?index=loosers