Python：如何填写 web 表单并获取生成的页面源

Question

I am trying to write a python script that will scrape http://www.fakenewsai.com/ and tell me whether or not a news article is fake news. I want the script to input a given news article into the website's url input field and hit the submit button. Then, I want to scrape the website to determine whether the article is "fake" or "real" news, as displayed on the website.

I was successful in accomplishing this using selenium and ChromeDriver , but the script was very slow (>2 minutes) and did not run on Heroku (using flask ). For reference, here is the code I used:

from selenium import webdriver
import time

def fakeNews(url):
  if url.__contains__("https://"):
    url = url[8:-1]
  if url.__contains__("http://"):
    url = url[7:-1]
  browser = webdriver.Chrome("static/chromedriver.exe")
  browser.get("http://www.fakenewsai.com")
  element = browser.find_element_by_id("url")
  element.send_keys(url)
  button = browser.find_element_by_id("submit")
  button.click()
  time.sleep(1)
  site = "" + browser.page_source
  result = ""
  if(site[site.index("opacity: 1")-10] == "e"):
    result = "Fake News"
  else:
    result = "Real News"
  browser.quit()
  return result

print(fakeNews('https://www.nytimes.com/2019/11/02/opinion/sunday/instagram-social-media.html'))

I have attempted to replicate this code using other python libraries, such as mechanicalsoup , pyppeteer , and scrapy . However, as a beginner at python , I have not found much success. I was hoping someone could point me in the right direction with a solution.

Answer 1

For the stated purpose, in my opinion it would be much more simple to analyze the website, understand it's functionality and then automate the browser behavior instead of the user behavior.

Try to hit F12 on your browser while on the website, open the Network tab, paste a URL on the input box and then hit submit, you will see that it sends a HTTP OPTIONS request and then a POST request to a URL. The server then returns a JSON response as a result.

So, you can use Python's request module ( docs ) to automate the very POST request instead of having a very complex code that simulates clicks and scrapes the result.

A very simple example you can build on is:

import json
import requests


def fake_news():
    url = 'https://us-central1-fake-news-ai.cloudfunctions.net/detect/'
    payload = {'url': 'https://www.nytimes.com/'}
    headers = {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.5',
               'Connection': 'keep-alive', 'Content-Length': '103', 'Content-type': 'application/json; charset=utf-8',
               'DNT': '1', 'Host': 'us-central1-fake-news-ai.cloudfunctions.net', 'Origin': 'http://www.fakenewsai.com',
               'Referer': 'http://www.fakenewsai.com/', 'TE': 'Trailers',
               'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0'}

    response_json = requests.post(url, data=json.dumps(payload), headers=headers).text
    response = json.loads(response_json)
    is_fake = int(response['fake'])

    if is_fake == 0:
        print("Not fake")
    elif is_fake == 1:
        print("Fake")
    else:
        print("Invalid response from server")


if __name__ == "__main__":
    fake_news()

PS: It would be fair to contact the owner of the website to discuss using his or her infrastructure for your project.

Answer 2

The main slowdown occurs on starting a chrome browser and locating the first URL. Note that you are launching a browser for each request. You can launch a browser on the initialization step and only do the automation parts per request. This will greatly increase the performance.