Python Web Scraping：美丽的汤

Question

I have a problem with the scraping of a web page. I'm trying to get the difference of points (Ex: +2,+1,...) between two teams but when I apply the find_all method it returns an empty list...

from bs4 import BeautifulSoup
from requests import get
url='https://www.mismarcadores.com/partido/Q942gje8/#punto-a-punto;1'
response=get(url)
html_soup=BeautifulSoup(response.text,'html.parser')


html_soup.find_all('span',class_='match-history-diff-score-inc')

Answer 1

The problem you have is the web content is being generated dynamically through JavaScript. As such, requests is unable to handle it, and so you'd be better off using something like Selenium .

EDIT: Per @λuser's suggestion, I've modified my answer to only use Selenium by searching for the elements you're looking for by XPath. Note that I used the XPath function starts-with() to get both match-history-diff-score-dec and match-history-diff-score-inc . Selecting only one of them was making you miss out on almost half of the relative score updates. This is why the output yields 103 results instead of 56.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.mismarcadores.com/partido/Q942gje8/#punto-a-punto;1")

table = driver.find_elements_by_xpath('//td//span[starts-with(@class, "match-history-diff-score-")]')

results = []
for tag in table:
    print(tag.get_attribute('innerHTML'))
print(results)

This outputs:

['+2', '+1', '+2', '+2', '+1', '+2', '+4', '+2', '+2', '+4', '+7', '+5', '+8', '+5', '+7', '+5', '+3', '+2', '+5', '+3', '+5', '+3', '+5', '+6', '+4', '+6', '+7', '+6', '+5', '+2', '+4', '+2', '+5', '+7', '+6', '+8', '+5', '+3', '+1', '+2', '+1', '+4', '+7', '+5', '+8', '+6', '+9', '+11', '+10', '+9', '+11', '+9', '+10', '+11', '+9', '+7', '+5', '+3', '+2', '+1', '+3', '+1', '+3', '+2', '+1', '+3', '+2', '+4', '+1', '+2', '+3', '+6', '+3', '+5', '+2', '+1', '+1', '+2', '+4', '+3', '+2', '+4', '+1', '+3', '+5', '+7', '+5', '+8', '+7', '+6', '+5', '+4', '+1', '+4', '+6', '+9', '+7', '+9', '+7', '+10', '+11', '+12', '+10']

Answer 2

Selenium might solve your problem but I suggest you track down the network from your browser and find the request that is generating the data you need. In your case it was d_mh_Q942gje8_es_1 .

I don't prefer Selenium because it is too heavy and makes your script slow. It was built for automated testing, not web scraping.

Here is my script using requests which undoubtedly runs faster than Selenium.

import requests
from bs4 import BeautifulSoup

url = 'https://d.mismarcadores.com/x/feed/d_mh_Q942gje8_es_1'

r = requests.get(url, headers={'x-fsign':'SW9D1eZo'}) # Got this from browser
soup = BeautifulSoup(r.text, 'html.parser')
diff_list = [diff.text for diff in soup.find_all('span',{'class' : 'match-history-diff-score-inc'})]
print(diff_list)

Output:

['+2', '+1', '+2', '+2', '+2', '+4', '+2', '+4', '+7', '+8', '+7', '+5', '+5', '+5', '+6', '+6', '+7', '+4', '+5', '+7', '+8', '+1', '+2', '+4', '+7', '+8', '+9', '+11', '+11', '+10', '+11', '+1', '+3', '+3', '+3', '+4', '+2', '+3', '+6', '+5', '+1', '+1', '+2', '+4', '+4', '+3', '+5', '+7', '+8', '+4', '+6', '+9', '+9', '+10', '+11', '+12']

Answer 3

If you check the page source (for example via view-source: in Chrome or Firefox, or by writing your html string to a file) you'll see that the element you are looking for (search match-history-diff-score-inc ) is not there. In fact, the rates are loaded dynamically using JS.