Python with Scrapy and Selenium get JavaScript generate content
Question
I'm using Python with Selenium and Firefox to get some content. The web site HTML is following:
<html> <meta http-equiv="X-UA-Compatible" content="IE=EmulateIE8"> <body leftmargin="0" topmargin="0" rightmargin="0" bottommargin="0" marginwidth="0" marginheight="0" onResize="resize();" bgcolor="#ffffff"> <iframe id="iframe" align="center" width="100%" height="100%" frameborder="0" marginWidth="0" marginHeight="0" src="" style="margin-top:1px;margin-left:1px;"></iframe> </body> <script> var eventTime = new Date(); function activeEvent() { eventTime = new Date(); } var height = 0; var width = 0; resize(); function setTitle(message) { document.title = message; } function resize() { height = document.body.clientHeight - 2; width = document.body.clientWidth - 2; if (height < 480) height = 480; if (width < 640) width = 640; document.getElementById("iframe").height = height; document.getElementById("iframe").width = width; } setInterval("resize()", 1000); document.getElementById("iframe").src = "http://xxxxxx.com:80/lhscm/framework/mainform/navui/nav3mainform.do?height=" + height + "&width=" + width; </script> </html>
But using Selenium driver.page_source can't get the src link generated HTML .
So, how can I get the content ?
1 answers
solution1
0 ACCPTED 2016-12-05 10:35:48
The body of your page seem to be completely located inside an iframe , so to get it you should switch to this frame. Try following:
driver.switch_to_frame('iframe')
source = driver.page_source
driver.switch_to_default_content()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
scrapy + selenium: <a> tag has no href, but content is loaded by javascript
Cannot select element from a javascript webpage using Python scrapy and Selenium
Python scrapy: how to retrive content that javascript dynamically fill into HTML
Get elements by javascript with Python Selenium
Issue scraping javascript generated content with Selenium and python
Scrapy Selenium clicking javascript link
Get WebElement text content with Selenium Driver in javascript
Get iframe content by using Javascript XPATH in Javascript executor for selenium webdriver
How can I detect if a page massively uses Javascript with Python, Scrapy and Selenium?
python selenium, get over javascript popup
Related Tags