无法使用Selenium单击按钮

Question

根据无法从pantip.com提取数据 ，我试图从pantip.com提取数据，包括所有评论和每个评论的回复。

我在获取每个评论的回复文本时遇到问题。 我使用硒单击按钮以获取文本内容。 但是，仅当我将页面滚动到按钮的位置时，它才有效。

如果我不滚动，这是一个错误。

WebDriverException: unknown error: Element <a href="javascript:void(0)" class="reply see-more">...</a> is not clickable at point (518, 507). Other element would receive the click: <select class="dropdown-jump">...</select>
  (Session info: chrome=71.0.3578.98)
  (Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 10.0.17134 x86_64)

有什么方法可以获取内部数据吗？ 我需要滚动到按钮吗？ 请给我建议。

import requests
import re
from bs4 import BeautifulSoup
from selenium import webdriver

chrome_path = r"C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe"
url='https://pantip.com/topic/38372443'
driver = webdriver.Chrome(chrome_path)
driver.get(url)

content=driver.page_source
soup=BeautifulSoup(content,'lxml')  

#Click all buttons
for div in soup.find_all("div", id = lambda value: value and value.startswith("reply-comment-")):
    xPath =  '''//*[@id="''' + str(div['id']) + '''"]/a''' 
    button = driver.find_element_by_xpath(xPath).click()


# Get all comments   
text = list()    
for div in soup.find_all("div", id = lambda value: value and value.startswith("comment-")):
    if len(str(div.text).strip()) > 1:
        text.append(str(div.text).strip())
driver.quit()

Answer 1

页面底部有一个固定的导航面板，因此，当您尝试单击按钮时，实际上是单击该面板中的元素，这就是引发Exception的原因……您可能需要

• 滚动到所需按钮
• 向下滚动

• 点击按钮查看回复

   from selenium.webdriver.common.keys import Keys for reply in driver.find_elements_by_xpath('//div[starts-with(@id, "reply-comment-")]/a'): driver.execute_script('arguments[0].scrollIntoView();', reply) reply.send_keys(Keys.DOWN) reply.click() 

Answer 2

要单击每个评论按钮并提取每个评论的回复文本，您不需要美丽的汤 ，仅硒就足够了。 为此，您必须将所需的元素滚动到视口中，并且可以使用以下解决方案：

• 代码块：

   from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC options = webdriver.ChromeOptions() options.add_argument("start-maximized") options.add_argument("disable-infobars") options.add_argument("--disable-extensions") driver= webdriver.Chrome(chrome_options=options, executable_path=r'C:\\Utility\\BrowserDrivers\\chromedriver.exe') driver.get("https://pantip.com/topic/38372443") comment_buttons = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.reply.see-more span.focus-txt"))) for button in comment_buttons: driver.execute_script("return arguments[0].scrollIntoView(true);", button) button.click() print("Comment button clicked") print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[class^='comment'][data-refcm^='comment'] div.display-post-story>a"))).get_attribute("innerHTML")) driver.quit() 

• 控制台输出：

   Comment button clicked Comment button clicked Comment button clicked Comment button clicked https://www.instagram.com/clintbondad . . . 

Answer 3

selenium可以使用execute_script("xxxx")执行脚本，例如：

script = """function getComments(){var comments=new Array();a = $('div.display-post-story');for (var i=0;i<a.length;i++){comments.push(a[i].innerText)};return comments;}"""
comments = driver.execute_script(script)
# then u can deal with all comments.