Cannot find element by xpath (JS activated website Selenium chrome)

Question

I am trying to scrape a website that uses Javascript. I have looked at similar questions on xpath in Selenium and they didn't really help. I tried using requests, but the Javascript doesn't fully load so I am using Selenium chrome driver.

I have tried both the full xpath, xpath, and class name and am unable to get the element. My code and the html are below.

site = 'https://finra-markets.morningstar.com/BondCenter/BondDetail.jsp?ticker=C614515&symbol=BBBY4144685'
browser = webdriver.Chrome(executable_path)
browser.get(site)
browser.find_elements_by_xpath('//*[@id="msqt_summary"]/div[2]/table/tbody/tr[1]/td[2]/span[@class="gr_text1"]')
browser.find_elements_by_xpath('//*[@id="no_border_no_center"]/div/div/div[1]/div/div/div[2]/div[2]/table/tbody/tr[1]/td[2]/span[@class="gr_text1"]')

<html xmlns="//www.w3.org/1999/xhtml"><head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
    <meta http-equiv="expires" content="12 Jan 2018">
    <title>Bond - Morningstar</title>
    <link href="//quotes.morningstar.com/bondq/static/common/css/reports.css" rel="stylesheet" type="text/css">     

</style>
<script type="text/javascript" src="//quotespeed.morningstar.com/js/qt_loader.js"></script><link rel="stylesheet" type="text/css" href="//finra-markets.morningstar.com/apis/quicktake/quicktake_finra.css"><script type="text/javascript" src="//finra-markets.morningstar.com/apis/quicktake/version_finra.js"></script>
<script type="text/javascript">window.onerror=function(msg){$("body").attr("JSError",msg);}</script>
</head>



<body id="no_border_no_center">
<div class="wrapper">
    <div class="gr_row_a5" id="msqt_summary"> 
               <div class="gr_colm_a2b gr_text1">
                   <table class="gr_table_b1" border="0" cellpadding="0" cellspacing="0" width="100%">
                       <tbody>
                           <tr valign="top">                               
                               <td class="gr_table_colm23">
                                   <h3 class="ms-gl-font10"> CUSIP</h3>                  
                                   <span class="gr_text1"> 075896AB6</span>
                               </td>                                
                            </tr>

Answer 1

The element is inside iframe.You need to switch it first to access the element.

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

site = 'https://finra-markets.morningstar.com/BondCenter/BondDetail.jsp?ticker=C614515&symbol=BBBY4144685'
browser = webdriver.Chrome(executable_path)
browser.get(site)
WebDriverWait(browser,10).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"ms-bond-detail-iframe")))    
print(browser.find_element_by_xpath('//*[@id="msqt_summary"]/div[2]/table/tbody/tr[1]/td[2]/span[@class="gr_text1"]').text)

To come out from iframe you need to use.

browser.switch_to.default_content()

Cannot find element by xpath (JS activated website Selenium chrome)

Question

1 answers

solution1
0 ACCPTED 2020-10-29 18:42:13

Cannot find element by xpath (JS activated website Selenium chrome)

Question

1 answers

solution1 0 ACCPTED 2020-10-29 18:42:13

solution1
0 ACCPTED 2020-10-29 18:42:13