使用python进行网页抓取。 从网站收集数据

Question

I am scraping some data from a website. I am writing a script that will automate it. In order to go to the next page I have to remove this disabled attribute from the button. I have tried many things, but nothing has helped me till now.

<button class="proceed" disabled="disabled" >Next</button>

Code:

binary = FirefoxBinary('/usr/lib/firefox/firefox')
driver = webdriver.Firefox(firefox_binary=binary)

driver.get("https://example.com/example/example")

my_url = Request('https://www.example.com/example/example', headers={'User-Agent': 'Mozilla/5.0'})
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")

user = raw_input("enter your starting point : ")

user_box = driver.find_element_by_class_name("search")
user_box.send_keys(user)
#user_box.autocomplete = "on"
#user_box.submit()

login_button = driver.find_element_by_css_selector("button.proceed")
#login_button.submit()

#print (login_button.text)

Answer 1

Use JavascriptExecutor to remove this disabled attribute from the button as follows :

login_button = driver.find_element_by_css_selector("button.proceed")
driver.execute_script("arguments[0].removeAttribute('disabled')", login_button)