无法使用Python scrapy和Selenium从javascript网页中选择元素

Question

I'm attempting to create an app that gathers some data. I'm using Python 2.7 with Scrapy and Selenium on Windows 10. I've done this with only a few web pages previously, however, I'm not able to select or click on a button from the following website.

https://aca3.accela.com/Atlanta_Ga/Default.aspx

Not able to click on the botton labeled "Search Permits/Complaints"

I've used the Chrome dev tools in order to inspect the XPaths, etc.

Here's the code that I'm using:

import scrapy
from selenium import webdriver

class PermitsSpider(scrapy.Spider):
  name = "atlanta"
  url = "https://aca3.accela.com/Atlanta_Ga/Default.aspx"

  start_urls = ['https://aca3.accela.com/Atlanta_Ga/Default.aspx',]

  def __init__(self):
    self.driver = webdriver.Chrome()
    self.driver.implicitly_wait(20)

  def parse(self, response):
    self.driver.get(self.url)

    time.sleep(15)
    search_button = self.driver.find_element_by_xpath('//*[@id="ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl"]')
    search_button.click()

When Running that code, I get the following error:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*@id="ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl"]"}

I've added all manor of sleeps and waits, etc in order to ensure that the page is fully loaded before attempting the selection. I've also attempted to select the Link Text, and other methods for selecting the element. Not sure why this method isn't working on this page where it works for me on others. While running the code, the WebDriver does open the page on my screen, and I see the page loaded, etc.

Any help would be appreciated. Thank you...

Answer 1

Target link located inside an iframe , so you have to switch to that frame to be able to handle embedded elements:

self.driver.switch_to_frame('ACAFrame')
search_button = self.driver.find_element_by_xpath('//*[@id="ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl"]')

You also might need to switch back with driver.switch_to_default_content()