I get nothing when trying to scrape a table

Question

So I want to extract the number 45.5 from here: https://www.myscore.com.ua/match/I9pSZU2I/#odds-comparison;over-under;1st-qrt

But when I try to find the table I get nothing. Here's my code:

import requests
from bs4 import BeautifulSoup
url = 'https://www.myscore.com.ua/match/I9pSZU2I/#odds-comparison;over-under;1st-qrt'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Raspbian Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36'}
response = requests.get(url, headers=headers)

soup = BeautifulSoup(response.content, 'html.parser')
text = soup.find_all('table', class_ = 'odds sortable')
print(text)

Can anybody help me to extract the number and store it's value into a variable?

Answer 1

You can try to do this without Selenium by recreating the dynamic request that loads the table.

Looking around in the network tab of the page, i saw this XMLHTTPRequest: https://d.myscore.com.ua/x/feed/d_od_I9pSZU2I_ru_1_eu

Try to reproduce the same parameters as the request.

To access the network tab: Click right->inspect element->Network tab->Select XHR and find the second request.

The final code would be like this:

headers = {'x-fsign' : 'SW9D1eZo'}

page = 
requests.get('https://d.myscore.com.ua/x/feed/d_od_I9pSZU2I_ru_1_eu', 
headers=headers)

You should check if the x=fisgn value is different based on your browser/ip.