HTML elements missing from Selenium page source, but can be found using BeautifulSoup
Question
So far I've looked at this , this .
I'm trying to parse an HTML source using Selenium. To make things easier (or so I thought), I took the HTML from the web page I'm going to parse and put it into a local HTML file.
BeautifulSoup has no problem viewing the HTML, but Selenium just doesn't see it for some reason.
HTML:
<html><head>
<meta http-equiv="Cache-control" content="no-cache">
<title>NOTICE TO CORES USERS</title>
</head>
<body>
<center>
<b>
<h1>Welcome</h1>
Hours of Operation<br><br>
Monday-Friday 6:00am - 10:00pm<br>
Saturday 8:00am - 6:00pm<br>
Sunday 12:00 noon - 6:00pm<br>
<br>
<br>
<h4><font color="purple"><p><b><u>CORES ORACLE UPGRADE</u></b><br>
<br>
<font color="BLACK">
<font color="RED">Due to a recent technical upgrade, CORES is experiencing a number of issues. We are aware of these issues and our teams are working to resolve them. Corporate Registry will provide updates when available. Corporate Registry apologizes for any inconvenience.
<br>
<br>
<font color="RED">Effective February 3, 2019, <font color="BLACK"> Corporate Registry will send annual return reminders by email to corporations, non-profit organizations, limited liability partnerships, and cooperatives where there is an email address on record.<br>
<br>
Annual return reminders will be emailed about two weeks before the annual return is due. The reminders will continue to be sent by regular mail
when there is no e-mail address on file or when there is a notice because the previous year's annual return has not been filed. Directors of Alberta corporations will continue to receive copies of the outstanding annual return notice by regular mail.
<br>
<br>
</font></font></font></font></p><h4><font color="BLACK"><font color="RED"><font color="RED"><font color="BLACK"><font color="purple"><p><b><u>EXTENDED OUTAGE DATES</u></b><br>
<font color="RED"></font></p><p align="CENTRE"><font color="RED">
FULL DAY outages to allow for technical preventive maintenance are as follows:
<br>
<br>
<font color="BLACK">Sunday, May 12, 2019<br>
<font color="BLACK">Sunday, June 9, 2019<br>
<font color="BLACK">Sunday, July 14, 2019<br>
<font color="BLACK">Sunday, August 11, 2019<br>
<font color="BLACK">Sunday, September 8, 2019<br>
<font color="BLACK">Sunday, October 13, 2019<br>
<font color="BLACK">Sunday, November 10, 2019<br>
<font color="BLACK">Sunday, December 8, 2019<br>
</font></font></font></font></font></font></font></font></font></p><p align="CENTRE"></p><h5><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">Updated: April 30, 2019</font></font></font></font></font></h5><p></p><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
<br>
<br>
<form action="cr_login.menu_frame" method="post">
<input type="hidden" name="p_default_menu" value="5">
<input type="hidden" name="p_system" value="CR">
<input type="hidden" name="p_accreditation" value="1">
<input type="hidden" name="p_spuid" value="30825">
<input type="hidden" name="p_userid" value="A02526">
<input type="submit" value="Continue">
</form>
</font></font></font></font></font></font></font></font></font></font></h4></font></h4></b></center><b><font color="purple"><font color="RED"><font color="RED"><font color="purple"><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
</font></font></font></font></font></font></font></font></font></b></body></html>
Python:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
chrome_options = Options()
chrome_options.binary_location = '/usr/bin/google-chrome'
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(chrome_options=chrome_options)
#driver.get('file:/c/Users/lanes/learning/product-repo/backend/functions/src/cores-scraper/oracle_upgrade.html')
driver.implicitly_wait(5)
driver.get('file:oracle_upgrade.html')
print('page source:', driver.page_source)
soup = BeautifulSoup(open('oracle_upgrade.html', 'r'), 'html.parser')
print('\nsoup:', soup)
driver.close()
Output:
page source: <html><head></head><body></body></html>
soup: <html><head>
<meta content="no-cache" http-equiv="Cache-control"/>
<title>NOTICE TO CORES USERS</title>
</head>
<body>
<center>
<b>
<h1>Welcome</h1>
Hours of Operation<br/><br/>
Monday-Friday 6:00am - 10:00pm<br/>
Saturday 8:00am - 6:00pm<br/>
Sunday 12:00 noon - 6:00pm<br/>
<br/>
<br/>
<h4><font color="purple"><p><b><u>CORES ORACLE UPGRADE</u></b><br/>
<br/>
<font color="BLACK">
<font color="RED">Due to a recent technical upgrade, CORES is experiencing a number of issues. We are aware of these issues and our teams are working to resolve them. Corporate Registry
will provide updates when available. Corporate Registry apologizes for any inconvenience.
<br/>
<br/>
<font color="RED">Effective February 3, 2019, <font color="BLACK"> Corporate Registry will send annual return reminders by email to corporations, non-profit organizations, limited liability partnerships, and cooperatives where there is an email address on record.<br/>
<br/>
Annual return reminders will be emailed about two weeks before the annual return is due. The reminders will continue to be sent by regular mail
when there is no e-mail address on file or when there is a notice because the previous year's annual return has not been filed. Directors of Alberta corporations will continue to receive copies of the outstanding annual return notice by regular mail.
<br/>
<br/>
</font></font></font></font></p><h4><font color="BLACK"><font color="RED"><font color="RED"><font color="BLACK"><font color="purple"><p><b><u>EXTENDED OUTAGE DATES</u></b><br/>
<font color="RED"></font></p><p align="CENTRE"><font color="RED">
FULL DAY outages to allow for technical preventive maintenance are as follows:
<br/>
<br/>
<font color="BLACK">Sunday, May 12, 2019<br/>
<font color="BLACK">Sunday, June 9, 2019<br/>
<font color="BLACK">Sunday, July 14, 2019<br/>
<font color="BLACK">Sunday, August 11, 2019<br/>
<font color="BLACK">Sunday, September 8, 2019<br/>
<font color="BLACK">Sunday, October 13, 2019<br/>
<font color="BLACK">Sunday, November 10, 2019<br/>
<font color="BLACK">Sunday, December 8, 2019<br/>
</font></font></font></font></font></font></font></font></font></p><p align="CENTRE"></p><h5><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">Updated: April 30, 2019</font></font></font></font></font></h5><p></p><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
<br/>
<br/>
<form action="cr_login.menu_frame" method="post">
<input name="p_default_menu" type="hidden" value="5"/>
<input name="p_system" type="hidden" value="CR"/>
<input name="p_accreditation" type="hidden" value="1"/>
<input name="p_spuid" type="hidden" value="30825"/>
<input name="p_userid" type="hidden" value="A02526"/>
<input type="submit" value="Continue"/>
</form>
</font></font></font></font></font></font></font></font></font></font></h4></font></h4></b></center><b><font color="purple"><font color="RED"><font color="RED"><font color="purple"><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
</font></font></font></font></font></font></font></font></font></b></body></html>
Question:
Why does Selenium not "see" the HTML in the body like Soup does?
2 answers
solution1
2 ACCPTED 2019-06-05 15:30:58
I was able to replicate your issue.
You are getting tricked because driver.get() wont really return an error if it is not able to load your file. Instead the driver.page_source will contain an almost empty document. I am not sure where your file is located, but i think you are just missing that a file URI should start with file:// and not file:
The following code worked for me:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('file:///Users/jimmy/src/stackoverflow/html-elements-missing-from-selenium-page-source-but-can-be-found-using-beautifu/oracle_upgrade.html')
print('page source:', driver.page_source)
driver.close()
You should not expect the exactly same output as the input since chrome will "fix" your html for you. For example if you forget <html> it will politely add it to the source for you without complaining.
solution2
1 2019-06-05 15:16:12
I cannot reproduce your issue using:
- Python 3.7.3
- Selenium 3.141.0 (according to
pip showcommand - Chromedriver 74.0.3729.6
- Chrome 74.0.3729.169
So I would recommend
upgrading to the latest
seleniumpackage version using pip like:pip install --upgrade seleniumcross-check your Chrome and ChromeDriver versions - it must be 100% match
alternatively you can try getting the page source via XPath expression
print(driver.find_element_by_xpath("/html").get_attribute("innerHTML"))
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Selenium Page Source is Missing Elements
Elements that are in page_source cannot be found when trying to scrape it using BeautifulSoup or Xpath
BeautifulSoup won't get the page source using selenium
retrieve text from the HTML source page using selenium with python
I can't find elements by Id inside html source page Selenium
Html missing when using View page source
BeautifulSoup + Python (Extract Specific HTML Tags from Page Source Code)
Extract links from html page using BeautifulSoup
Extract data from html page using Beautifulsoup
python selenium, can't find elements from page_source while can find from browser
Related Tags