我无法使用 selenium 和美丽的汤来抓取表数据

Question

I've went as far as I can go but I can't seem to scrape data from a table. I've searched through stackoverflow for answers but nothing seems to work. Essentially the table comes up empty or I simply can't find elements within the table. I'm working with a table from yahoo's daily fantasy webpage.

NOTE: the current web address used will likely change week to week so it may not be a valid address in the future.

Current Code:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait

driver = webdriver.Chrome()
driver.get("https://sports.yahoo.com/dailyfantasy/contest/5416455/setlineup")

response = wait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME,"data-tst-player-id")))
driver.quit

soup = BeautifulSoup(response, 'lxml')
with open('test.txt','w', encoding='utf-8') as f_out:
    f_out.write(soup.prettify())

Answer 1

There is no element with the class-name or id that you are providing in the line

response = wait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME,"data-tst-player-id")))

there is however, some tags with the attribute 'data-tst', so you can use that to make sure your page has loaded, and on this line

driver.quit

you are doing nothing, you have to call the function driver.quit(). working code:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait

driver = webdriver.Chrome()
driver.get("https://sports.yahoo.com/dailyfantasy/contest/5416455/setlineup")
wait(driver, 1).until(EC.presence_of_element_located((By.CSS_SELECTOR,"[data-tst]")))
response=driver.page_source
driver.quit()

soup = BeautifulSoup(response, 'lxml')
with open('test.txt','w', encoding='utf-8') as f_out:
    f_out.write(soup.prettify())