使用 Python 请求模拟单击“显示更多”按钮

Question

I am not sure what code to use for clicking the show more button. I want to get a list of university who are doing certain topic. below is one of the websites

http://www.sciencedirect.com/science/article/

your helps will be true appreciated

Thanks

Answer 1

You shouldn't have to simulate, in Python, an actual "click" of the "show more" button to accomplish web-scraping.

"Show more" buttons in websites are usually tied to some JavaScript that either reveals a hidden element already in the HTML (see Bootstrap's collapse class for a typical example) or fires off a request to some web service (eg a REST API ) for information to insert in the DOM .

Either way, you can scrape that data. For the former, find the hidden element in the DOM (view the page's source [ Ctrl + U ] and search the HTML [ Ctrl + F ]), and use your typical webscraping tools. For the latter, use something like Google Dev Tools' Network tab to inspect the API request when you click "show more" and then try to replicate that request with Python.

In the specific example you've given, it appears the data you want is stored in an HTML <script> tag as a JSON object. Search the HTML for the word "affiliation".

Answer 2

You'll have to select a different tool to press a button. One possible solution is Selenium, which can tell the browser to press the button. The following example clicks the show more button.

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def executeTest():
    global driver
    driver.get('http://www.sciencedirect.com/science/article/pii/S2211926417300024')
    time.sleep(7)
    element = driver.find_element_by_xpath('//*[@id="app"]/div/div/div/section/div/div[2]/article/div[2]/button')
    element.click()
    time.sleep(3)

def startWebDriver():
    global driver
    options = Options()
    options.add_argument("--disable-infobars")
    driver = webdriver.Chrome(chrome_options=options)

if __name__ == "__main__":
    startWebDriver()
    executeTest()
    driver.quit()

Answer 3

I just got round a similar problem by reading Michael Crenshaw's answer above. Here's what worked for me:

1. Load the page you want to scrape.
2. Go to inspect and select network.
3. Now click the show more button.

You should now see in the network tab the exact url where the request is being made to. It's a lot easier if you don't open the network tab until the page has already loaded - that way the only information in the tab is the request it makes when you click show more.

I then just added a few lines to my code like this:

page_source = response.text
if "Show More" in page_source:

And then after this I added my scraping function to that while getting it to iterate through the url structure. There's a good post on how to do that on scrapy here - https://blog.scrapinghub.com/2016/06/22/scrapy-tips-from-the-pros-june-2016 .

Hope this helps.