PYTHON : Selenium loop selenium.common.exceptions.TimeoutException

Question

I'm trying to scrape product details from each individual product page. I try to append an integer in between xpath for 'elements'. The for i in range loop run without problem for j = 1 , print getproductname but throw error for j = 2 .

How do you fix the loop? I couldn't figure out what seems to be the issue here:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

CHROMEDRIVER_PATH = '/Users/reezalaq/PycharmProjects/wholesale/driver/chromedriver'
chrome_options = webdriver.ChromeOptions()
driver = webdriver.Chrome(CHROMEDRIVER_PATH, options=chrome_options)
chrome_options.accept_untrusted_certs = True
chrome_options.assume_untrusted_cert_issuer = True
chrome_options.headless = False

driver.get('https://www.skinnymixes.com/collections/skinny-syrups')
for i in range(1,20):
    j = str(i)
    print(j)
    elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '#collection-content > div.collection-listing > div > div.row.product-list.text-center.mb-3.in-view.in-view--active.in-view--loaded > div:nth-child('+ j +') > a')))
    print(elements)
    for element in elements:
        url = element.get_attribute('href')
        print(url)
        #open new tab with specific url
        driver.execute_script("window.open('" +url +"');")
        #switch to new tab
        driver.switch_to.window(driver.window_handles[1])
        getproductname = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@id="shopify-section-product"]/div/div[1]/div/div[2]/h1')))
        print(getproductname.text)

Traceback (most recent call last):
  File "/Users/reezalaq/PycharmProjects/legasimall/skinny/getproducts.py", line 70, in <module>
    elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '#collection-content > div.collection-listing > div > div.row.product-list.text-center.mb-3.in-view.in-view--active.in-view--loaded > div:nth-child('+ j +') > a')))
  File "/Users/reezalaq/PycharmProjects/legasimall/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

Answer 1

The problem with your code is that

• it opens a product in a new window, switches driver to it and then it scrapes the information

• but after that it is NOT switching the driver back to the original window.

So, basically at the end of the loop you need to switch it back to original window, ie,

either,

driver.switch_to.default_content()

or, save the original window handle before switching to new window and use that to switch back the driver to it.