Python Selenium: Getting dynamic content within iframe

Question

I am trying to scrape the available apartment listings from the following webpage: https://3160599v2.onlineleasing.realpage.com/

I am using the Python implementation of Selenium, but so far I haven't found an effective solution to programmatically get the content. My most basic code is the following, which currently just returns the non-dynamic HTML source code:

from selenium import webdriver

driver = webdriver.Chrome('/path_to_driver')
driver.get('https://3160599v2.onlineleasing.realpage.com/')
html = driver.page_source

The returned html variable does not contain the apartment listings I need.

If I 'Inspect' the element using Chrome's built-in inspect tool, I can see that the content is within an un-classed iframe: <iframe frameborder="0" realpage-oll-widget="RealPage-OLL-Widget" style="width: 940px; border: none; overflow: hidden; height: 2251px;"></iframe>

Several children down within this iframe you can also see the div <div class="main-content"> which contains all the info I need.

Other solutions I have tried include implementing an explicit WebDriverWait:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CLASS_NAME, 'main-content')))

I get a TimeoutException with this method as the element is never found.

I also tried using the driver.switch_to.frame() method, with no success.

The only steps that have actually allowed me to get the apartment listings out of the webpage have been (using Chrome):

1. Manually right-click on an element of the listings within the webpage
2. Click Inspect
3. Find the div 'main-content'
4. Manually right-click on this div and select Copy -> Copy Element

This is not an effective solution since I'm seeking to automate this process.

How can I get this dynamically generated content out of the webpage in a programatic way?

Answer 1

Try to use below code to switch to iframe:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait

wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_xpath('//iframe[@realpage-oll-widget="RealPage-OLL-Widget"]')))

Also note that method that allows to switch to static iframe is switch_to.frame() , but not switch-to.frame()

Answer 2

You can not directly see the content which is in the iframe. You need to change frame. You can do this by firstly selecting 'iframe element' and then switching to it with driver.switch_to.frame() function.

iframe = driver.get_element_by_id('iframe')
driver.switch_to.frame(iframe)

After that you can access the iframe's content.

Alternatively, you can take the source attribute of iframe then going to that page with selenium. In the end, iframe content is another html page.