元素未响应 Python 请求

Question

I would to scrape the last odds in archive from this page https://www.betexplorer.com/soccer/estonia/esiliiga/elva-flora-tallinn/Q9KlbwaJ/ but I can't get it with requests. How can I get it without interact with Selenium ? To trigger the archive odds page in the Developer Tools I need to hover on the odd.

Code

 url = "https://www.betexplorer.com/archive-odds/4l4ubxv464x0xc78lr/14/"
 headers = {
            "Referer": "https://www.betexplorer.com",
                    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'
               }
Json = requests.get(url, headers=headers).json()

Answer 1

As the site is being loaded by JavaScript, requests doesn't work. I have used selenium to load the page, extract the complete source code after everything is loaded.

Then used beautifulsoup to create a soup object to get required data.

From the source code you can see that the data-bid of the <tr> are what are being passed to get the odds data.

I extracted all the data-bid and passed them to the URL you've provided at the very end of your question one by one.

This code will get all the odds data in JSON format

import time
from bs4 import BeautifulSoup
import requests
from selenium import webdriver

base_url = 'https://www.betexplorer.com/soccer/estonia/esiliiga/elva-flora-tallinn/Q9KlbwaJ/'
driver = webdriver.Chrome()
driver.get(base_url)

time.sleep(5)

soup = BeautifulSoup(driver.page_source, 'html.parser')
t = soup.find('table', attrs= {'id': 'sortable-1'})
trs = t.find('tbody').findAll('tr')

for i in trs:
    data_bid = i['data-bid']
    url = f"https://www.betexplorer.com/archive-odds/4l4ubxv464x0xc78lr/{data_bid}/"
    headers = {"Referer": "https://www.betexplorer.com",'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'}
    Json = requests.get(url, headers=headers).json()
    
    # Do what you wish to do withe JSON data here....