无法在python中使用selenium在网页中加载“显示更多”？

Question

I am trying to load "Show More" using selenium in python. However, the code is running and executing fine but I am getting 'Message' and 'complete'(this I understand) in python console. Also, the webpage I am trying to open is opening but as soon as I try to scroll down it is showing a window, "This Safari window is remotely controlled by an automated test." Even though I click 'continue session' and again try to scroll down the page, it again shows the same window.

I am working on MacBook​ Air and using Safari as a browser.

from selenium import webdriver

import time

browser = webdriver.Safari(executable_path = '/usr/bin/safaridriver')

browser.get('https://www.nytimes.com/search?endDate=20190331&query=cybersecurity&sort=newest&startDate=20180401')

Show_more_xpath = '//*[@id="site-content"]/div/div[2]/div[3]/div/button'

while True:
    try:
        show_more = browser.find_elements_by_xpath('//button[@type="button"], and contains(.,"Show More")')
        time.sleep(2)
        show_more.click()
        time.sleep(5)
    except Exception as e:
        print(e)
        break
print("Complete")
time.sleep(10)

browser.quit()

I want to load "Show More" automatically to get all the entries. The ultimate goal is to scrape the page using beautifulsoup.

Answer 1

driver.find_element_by_xpath（“（.// * [normalize-space（text（））和normalize-space（。）='有搜索反馈吗？让我们知道你的想法。']）[1] / follow :: button [1]“）。单击（）

Answer 2

To click on Show More Button do the followings.

Click on I Accept button first and then use WebdriverWait and element_to_be_clickable to wait for for element to be clickable and then the use action class to click on the specific button.Try to below code hope this will work for you.

browser.get('https://www.nytimes.com/search?endDate=20190331&query=cybersecurity&sort=newest&startDate=20180401')

browser.find_element_by_xpath("//button[contains(.,'I Accept')]").click()

while True:
    try:
        show_more = WebDriverWait(browser, 5).until(expected_conditions.element_to_be_clickable((By.XPATH, '//button[@type="button"][contains(.,"Show More")]')))
        ActionChains(browser).move_to_element(show_more).click().perform()
    except Exception as e:
        print(e)
        break

print("Complete")
time.sleep(10)

browser.quit()

You need to have following imports to execute above code.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.action_chains import ActionChains