Selenium web 抓取：如何在响应式网站上查找元素

Question

I am on a project that needs to get some information from a website and i wanted to automnatizate this process, but i am a begginer with the technology to achive this.

I found this library(Selenium) for python on the internet and i thougth it maight be a solution.

I succeded the firsts steps (Accepting cookies, locating the "Access" button and clicking on it). the problem comes when the "Access" button is clicked. It shows a little form to input the user and password, but i couldn't found them using the driver.find_... methods, so i began to look for the elements on the html document and it seems that this form is injected after clicking on the "Accept" button.

Is there any strategy to find the input elements for introducing the user and password after clicking the button?

HTML code of the website without having clicked de "Access" button

HTML code after clicking on the "Access" button

Thank you

The website link is: https://www.bbva.es/personas/experiencias/bbva-valora/analiza-vivienda.html

Answer 1

<iframe class="iframe__base" title="People Login" id="tab-personas-iframe" width="100%" style="max-width: 100%; overflow: hidden; height: 384px;" src="/nimbus/login.html?conf=net/login&amp;http_contactid=04a341b5-c0ce-454a-8854-be6b12299c9a" scrolling="no">Your browser does not support iframes</iframe>

Your elements are inside an iframe.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC


wait=WebDriverWait(driver, 40)
driver.get('https://www.bbva.es/personas/experiencias/bbva-valora/analiza-vivienda.html')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,".cookiesgdpr__acceptbtn.btn__basic.btn__medium-blue"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,".header__actions__item__link.header__actions--menu.header__access"))).click()

So here you have to switch to the iframe and then find element

wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID,"tab-personas-iframe")))
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@data-testid='login-form-user']"))).send_keys("USER")
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@data-testid='login-form-password']"))).send_keys("PASSWORD")
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@data-testid='login-form-submit']"))).click()

To get where you want after you click the accept popup and the acceso item you just switch frames and send key.