How to scrape in Python Selenium?

Question

If you visit this site, https://www.premierleague.com/match/66686 and then press stats tab, you will see several information about the match. How am I supposed to scrape the Possession for both teams?

This did not work.

stats = driver.find_element(By.XPATH, '//*[@id="mainContent"]/div/section[2]/div[2]/div[2]/div[1]/div/div/ul/li[3]')
stats.click()
    posHome = driver.find_element(By.XPATH,'//body[1]/main[1]/div[1]/section[2]/div[2]/div[2]/div[2]/section[3]/div[2]/div[2]/table[1]/tbody[1]/tr[1]/td[1]')
    print(posHome.text)
    posAway = driver.find_element(By.XPATH,'//*[@id="mainContent"]/div/section[2]/div[2]/div[2]/div[2]/section[3]/div[2]/div[2]/table/tbody/tr[1]/td[3]')
    print(posAway.text)

Please let me know how to solve this issue and thanks!

Answer 1

To print the Possession for both teams you need to induce WebDriverWait for the visibility_of_element_located() and you can use the following locator strategies :

• Code Block:

   driver.get("https://www.premierleague.com/match/66686") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Accept All Cookies']"))).click() WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[text()='Stats']"))).click() print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tbody.matchCentreStatsContainer>tr>td>p"))).text) print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tbody.matchCentreStatsContainer>tr>td:nth-child(3)>p"))).text) driver.quit()

• Console Output:

   33.9 66.1

• Note : You have to add the following imports:

   from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python