如何使用 selenium 抓取登录网站

Question

from selenium import webdriver

from time import sleep

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.3"
}

driver= webdriver.Chrome('C:\Program Files (x86)\chromedriver.exe')
driver.get('https://www.counselingcalifornia.com/Find-a-Therapist')
driver.find_element_by_id("language_field").click("option",value="ENG")
driver.find_element_by_id(searchBtn).click()
print("successfully")

I am trying to scrape data but first Click on Language, select English, then click advanced search but they will give me error these is website https://www.counselingcalifornia.com/Find-a-Therapist

Answer 1

The reason is iframe.

language drop down is in iframe. In Selenium automation, if the webelements are wrapped inside an iframe, we should always switch to iframe first then we can interact with the elements.

Code :

driver = webdriver.Chrome(driver_path)
driver.maximize_window()

wait = WebDriverWait(driver, 30)

driver.get("https://www.counselingcalifornia.com/Find-a-Therapist")

wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[id$='IFrame_htmIFrame']")))
select = Select(wait.until(EC.visibility_of_element_located((By.ID, "language_field"))))
select.select_by_value('ENG')

wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#searchBtn"))).click()

Imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select

Answer 2

If there was error related to the chromedriver version, then please install the latest version of the driver. You have a syntax error in your code. Include your searchBtn id in double-quotes.

driver.find_element_by_id("searchBtn").click()

from selenium import webdriver

from time import sleep

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.3"
}

driver= webdriver.Chrome('C:\Program Files (x86)\chromedriver.exe')
driver.get('https://www.counselingcalifornia.com/Find-a-Therapist')
driver.find_element_by_id("language_field").click("option",value="ENG")
driver.find_element_by_id("searchBtn").click()
print("successfully")