Selenium Python: Unable to scroll and click on Load more stories button

Question

I am new to Selenium and would appreciate advice. I am trying to click on the load more stories button on NPR's website ( https://www.npr.org/sections/news/ ). My code seems good, but when I run it more stories are never loaded. I don't get an error at the same time the code just doesn't behave as expected.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox(executable_path='/home/myname/projects/myproject/geckodriver')
# implicit wait for 5 seconds
driver.implicitly_wait(5)
#driver.maximize_window()
driver.get('https://www.npr.org/sections/news/')

element = driver.find_element(By.XPATH, '/html/body/main/div[2]/section/div[7]/button')
driver.execute_script("arguments[0].scrollIntoView();", element)
driver.execute_script("arguments[0].click();", element)

Answer 1

The webpage npr have a infinitescrollwrap . As a demonstration to click() thrice on Load more stories you can use the following Locator Strategies :

• Code Block:

   driver.get("https://www.npr.org/sections/news/") WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.global-modal__dismiss"))).click() for i in range(3): driver.execute_script("arguments[0].scrollIntoView();", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button.options__load-more")))) WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.options__load-more"))).click() print("Load more stories clicked")

• Console Output:

   Load more stories clicked Load more stories clicked Load more stories clicked

• Note : You have to add the following imports:

   from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC