使用 python、BeautifulSoup、Selenium 从表中抓取动态数据

Question

I would like to scrape all the url links associated with the soccer games included in the table in this website .

Here is the code:

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Firefox()
url = 'https://www.coteur.com/cotes-foot.php'
driver.get(url)

fixture1 = driver.find_element_by_xpath("/html/body/div[3]/div/div[2]/div/div/div[2]/div/table/tbody/tr[3]/td[3]/a")
print(fixture1.text)

links = []
i = 3
while i <= 6:
    fixture = driver.find_element_by_xpath("/html/body/div[3]/div/div[2]/div/div/div[2]/div/table/tbody/tr[" + str(i) + "]/td[3]/a")
    links.append(fixture)
    i = i + 3

print(links)

driver.close()

When I scrape one match it returns the data I'm expecting. However, when I tried to make a loop to get all the soccer games I run into a problem.

Here is the result of the code:

Betis Seville - Granada 74 Cf
[<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="0199958a-4d31-4a21-9856-8f8c3cc8ee05", element="158fcdaf-501f-41a4-9550-8a42543acc22")>, <selenium.webdriver.firefox.webelement.FirefoxWebElement (session="0199958a-4d31-4a21-9856-8f8c3cc8ee05", element="74e67896-fccb-48da-8eef-bbf8d9a6f3b3")>]

I wanted to get the first element, but I don't get what I was expecting.

Answer 1

This works well

    from selenium import webdriver

    driver = webdriver.Firefox()
    driver.get("https://www.coteur.com/cotes-foot.php")

    links = driver.find_elements_by_xpath('//a[contains(@href, "match/cotes-")]')

    data = [l.text for l in links]

    print(data)

Answer 2

I tried your code, here is the result:

File "./coteur2.py", line 17
    data = [l.text for l in links]
    ^
IndentationError: unexpected indent

I prefer to use this way:

links = driver.find_elements_by_xpath('//a[contains(@href, "match/cotes-")]')

n = 0
while n < len(links):
   links[n] = links[n].text
   n = n + 1

print(links)

Thanks for your help