硒抓取javascript

Question

I'm planning on making a website that scrapes a lot of daily updated URLS (JavaScript) from many websites. I did some research and found selenium, already made some code to extract a URL from a website

from selenium import webdriver
chrome_path = r"C:\Users\hessien\Desktop\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("http://example.com")
driver.find_element_by_xpath("""//*[@id="header"]/div/div[2]/div[3]/ul/li/label/a""").click()
element = driver.find_element_by_xpath("""//*[@id="s"]""")
element.send_keys("example")
driver.find_element_by_xpath("""//*[@id="searchform"]/button/span""").click()
driver.find_element_by_xpath("""//*[@id="contenedor"]/div/div[2]/div[1]/div[2]/article/div[2]/div[1]/a""").click()
driver.find_element_by_xpath("""//*[@id="playex"]/div[1]""").click()
elem = driver.find_element_by_xpath("""//*[@id="mediaplayer_media"]/video""").get_attribute("src");
print elem

but after some searches I found out that selenium mainly used as a testing framework not for scraping and crawling!.. my question is can selenium do the work? if yes, how to execute the python code in an HTML button? I'm also using Django. if no, could you recommend anything that can do the task?

Answer 1

If you really want to make a scraper i recommend you to use Beautiful soup, which is a Python library for pulling data out of HTML and XML files. you can integrate the python script with django which can be triggered on a click. following is the link.

https://pypi.python.org/pypi/beautifulsoup4