How can i get html from selenium.webdriver in Python?

Question

Thank you for your attention,and sorry for my poor english.

I am working at get html from http://flvxz.com/ when i put some words in the input box and click.it is like :

• load the Yahoo homepage
• search for "something" ( input someword and click button )
• get the return html

my websit return the message by javaScrip

this is my code

browser = webdriver.Firefox()

browser.get('http://flvxz.com/')

input_box = browser.find_element_by_id("videoPageURI")  # Find the search box
button = browser.find_element_by_id("fetchButton")  # find the button

input_box.send_keys(url)  # input some string
button.click()  # click

then,how can i get the new html? i try as below,not work.

html = browser.find_element_by_tag_name('html').text();

My questions :

1. How can i get html?
2. Is there better ways or tools to do this jop(i do not like it open the window ,but the i like the way it control the browser)?

Thank you a lot,and i am improving my english...

Answer 1

point 1. how to get the html code:

first of all, you might need to wait until the page is fully loaded (see this article )

Then all you need to do is access the page_source attribute:

html = browser.page_source

point 2. is there a better way to do this:

as far as I know, that is the correct way to do it, with Selenium. There are alternative ways to find the elements you need in the page, but it would not change much in terms of performance.

If you do not need to see the actual page being displayed, I suggest to use a headless browser (eg PhantomJS) instead of Firefox

driver=webdriver.PhantomJS('your pahtomjs exe file location')

See this SO QA for more info and this tutorial on how to setup PhantomJS with python