How to get some data in real time from a website using python?

Question

I want to fetch som data from website

https://web.sensibull.com/optionchain?expiry=2020-03-26&tradingsymbol=NIFTY

I am using beautifulsoup library to fetch this data, and have tried the following code:

import requests

import urllib.request

import time

from bs4 import BeautifulSoup

url = 'https://web.sensibull.com/optionchain?expiry=2020-03-26&tradingsymbol=NIFTY'

response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')

b = soup.find("div", {"class": "style__AtmIVWrapper-idZNMX kUMMRI"})

print(b)

But it shows "None" as the output.

Although there is only one class of this name in the full HTML code, but I also tried this:

for b in soup.find_all('div', attrs={'class':'style__AtmIVWrapper-idZNMX kUMMRI'}):

    print(b.get_text())

    print(len(b))

But it doesn't work.

Also tried soup.find("div") But it does not shows the required div tag in the output, maybe due to nested divs present.

Unable to fetch this data and proceed with my work. Please help.

Answer 1

Could be a syntax problem try with soup.find_all("div", class_="style__AtmIVWrapper-idZNMX kUMMRI") or just soup.find("div", class_="style__AtmIVWrapper-idZNMX kUMMRI")

If interested in webscraping and bs4 take a look at the documentation https://www.crummy.com/software/BeautifulSoup/bs4/doc/#find

Answer 2

If you are looking for code. This might help:-

from selenium import webdriver 
import time
webpage = 'https://web.sensibull.com/optionchain?expiry=2020-03-26&tradingsymbol=NIFTY'
driver = webdriver.Chrome(executable_path='Your/path/to/chromedriver.exe') 
driver.get(webpage)
time.sleep(10)
nifty_fut = driver.find_element_by_xpath('//*[@id="app"]/div/div[4]/div[2]/div[3]/div/div/div[2]/div[1]/div[1]/div/button/span[1]/div[1]')
print(nifty_fut.text)
atm_iv = driver.find_element_by_xpath('//*[@id="app"]/div/div[4]/div[2]/div[3]/div/div/div[2]/div[1]/div[2]')
print(atm_iv.text)
driver.quit()