can not switch to a new page using python selenium webdriver?

Question

I need to scrape the CSV result of the website ( https://requestmap.webperf.tools/ ). But I can't.

This is the process I need to do:

1- load the website ( https://requestmap.webperf.tools/ )

2- enter a new website as an input (for example, https://stackoverflow.com/ )

3- click submit

4- in the new page which opens after the submit, download the csv file at the end of the page

But I think the driver has the main page and doesn't switch to the new page. That's why I can't download the CSV file.

Would you please tell me how to do it? here is my code:

options = webdriver.ChromeOptions()
options.add_argument('-headless')
options.add_argument('-no-sandbox')
options.add_argument('-disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=options)
driver.get("https://requestmap.webperf.tools/")


driver.find_element_by_id("url").send_keys("https://stackoverflow.com/")
driver.find_element_by_id("submitter").click()

I have tested different ways to solve my issue: First of all: I got this code from here : But it doesn't work.

window_after = driver.window_handles[1]                           
driver.switch_to.window(window_after)

I also tried this or this , but they are not working as well.

# wait to make sure there are two windows open
# it is not working
WebDriverWait(driver, 30).until(lambda d: len(d.window_handles) == 2)

# switch windows
# it is not working
driver.switch_to_window(driver.window_handles[1])

content = driver.page_source
soup = BeautifulSoup(content)
driver.find_elements_by_name("Download CSV")

So How can I get this issue solved?

Is there any other way in python to do so and switch to a new windows?

Answer 1

The problem is your waiting condition is incorrect. after you click submit the window changes but it doesn't open a new window.

a better option is to just wait for the page with the.csv file to load and then download it.

my solution:

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


options = webdriver.ChromeOptions()
options.add_argument('-headless')
options.add_argument('-no-sandbox')
options.add_argument('-disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=options)
driver.get("https://requestmap.webperf.tools/")

driver.find_element_by_id("url").send_keys("https://stackoverflow.com/")
driver.find_element_by_id("submitter").click()

WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.XPATH, "//a[contains(text(), 'Download CSV')]")))
driver.find_element(By.XPATH, "//a[contains(text(), 'Download CSV')]").click()

so as you can see i am waiting until the "Download CSV" is located and then i press it.

Also notice that the download directory will be the chrome default but you can change it by adding:

options.add_argument("download.default_directory=path")