我该如何添加多线程呢？

Question

I don't know how to web scrape that much, I wrote this code but it is running really slowly, this code is used to get the search results from a google chrome query. I want to try to add multithreading but I don't really know how. Can somebody tell me how to multithread? Also which function am I supposed to multithread?

import urllib
import requests
from bs4 import BeautifulSoup
from multiprocessing import Pool
# desktop user-agent

def get_listing(url):
    headers = {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'}
    html = None
    links = None

    r = requests.get(url, headers=headers, timeout=10)

    if r.status_code == 200:
        html = r.text
        soup = BeautifulSoup(html, 'lxml')
        listing_section = soup.select('#offers_table table > tbody > tr > td > h3 > a')
        links = [link['href'].strip() for link in listing_section]
    return links

def scrapeLinks(query_string):
    USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"

    query = query_string
    query = query.replace(' ', '+')
    URL = f"https://google.com/search?q={query}"

    headers = {"user-agent": USER_AGENT}
    resp = requests.get(URL, headers=headers)

    if resp.status_code == 200:
        soup = BeautifulSoup(resp.content, "html.parser")
        results = []
        for g in soup.find_all('div', class_='r'):
            anchors = g.find_all('a')
            if anchors:
                link = anchors[0]['href']
                title = g.find('h3').text
                item = {
                    "title": title,
                    "link": link
                }
                results.append(item)
        return results

def getFirst5Results(query_string):
    list = scrapeLinks(query_string)
    return [list[0]["link"], list[1]["link"], list[2]["link"], list[3]["link"], list[4]["link"]]

Answer 1

Few things about multithreading

• You can use it for the code that required network calls. For instance, invoking an api.
• When the code would run for a longer duration of time, and you want to run the process in the background.
• In the case you've stated the web scraping is a long running tasks, as it involves network call to google api and parsing of the results after we get the results back. Assuming that you're using scrapeLinks function for scraping. Here's some code:

import threading t1 = threading.Thread(target = scrapeLinks, args = (query_string,) t1.start()

In order to retrieve results from the thread use: t1.join()