使用 bs4 进行网页抓取以检索文本值

Question

import pandas as pd
import requests
from bs4 import BeautifulSoup
import re

url = 'https://www.betexplorer.com/odds-movements/soccer/'

res = requests.get(url)
soup = BeautifulSoup(res.content, "lxml")
times = soup.select('span.table-main__time') #good
matches = soup.find_all("td",class_ ="table-main__tt")

odds = soup.find_all("td",class_ ="table-main__odds")

刮擦的所需目标是data-odd="value"之后的值

我试过odds[0].a.text无济于事

关于如何使用bs4提取这些值有什么建议吗？

Answer 1

检查文档的属性并使用tag.get('attr')如果您不确定attr是否已定义，就像使用 Python dictionary一样。

例子

import requests
from bs4 import BeautifulSoup

url = 'https://www.betexplorer.com/odds-movements/soccer/'
soup = BeautifulSoup(requests.get(url).content)

data = []

for m in soup.select('#odds-movements tr:has(.table-main__tt)'):
    data.append({
        'match':m.a.text,
        'time':m.span.text,
        'odds':[o.get('data-odd') for o in m.select('a[data-odd]')],
        'adapt':'the concept to add additional scraped information'
        
    })
data

Output

[{'match': 'Monterrey W - U.N.A.M.- Pumas W',
  'time': '04:00',
  'odds': ['1.32', '5.10', '6.56', '6.56'],
  'adapt': 'the concept to add additional scraped information'},
 {'match': 'Santamarina - Gimnasia Mendoza',
  'time': '01:30',
  'odds': ['3.37', '3.37', '2.85', '2.25'],
  'adapt': 'the concept to add additional scraped information'},
 {'match': 'Club America W - Pachuca W',
  'time': '02:00',
  'odds': ['1.77', '3.85', '3.53', '3.53'],
  'adapt': 'the concept to add additional scraped information'},...]