使用请求和BeautifulSoup的Python多线程

Question

我在写网络刮板。 我本可以使用scrapy但决定从头开始编写它，以便我可以练习。

我创建了一个刮板，可以使用请求和BeautifulSoup成功运行。 它浏览大约135页，每页12个项目，获取链接，然后从链接目标中获取信息。 最后，它将所有内容写入CSV文件。 它仅捕获字符串，并且目前不下载任何图像或类似内容。

问题？ 很慢 仅从一页的内容中获取所有内容大约需要5秒钟，因此时间135约为11分钟。

所以我的问题是我该如何在代码中实现线程化，以便更快地获取数据。

这是代码：

import requests
from bs4 import BeautifulSoup
import re
import csv


def get_actor_dict_from_html(url, html):
    soup = BeautifulSoup(html, "html.parser")

    #There must be a better way to handle this, but let's assign a NULL value to all upcoming variables.
    profileName = profileImage = profileHeight = profileWeight = 'NULL'

    #Let's get the name and image..
    profileName = str.strip(soup.find('h1').get_text())
    profileImage = "http://images.host.com/actors/" + re.findall(r'\d+', url)[0] + "/actor-large.jpg"

    #Now the rest of the stuff..
    try:
        profileHeight = soup.find('a', {"title": "Height"}).get_text()
    except:
        pass
    try:
        profileWeight = soup.find('a', {"title": "Weight"}).get_text()
    except:
        pass

    return {
        'Name': profileName,
        'ImageUrl': profileImage,
        'Height': profileHeight,
        'Weight': profileWeight,
        }


def lotta_downloads():
    output = open("/tmp/export.csv", 'w', newline='')
    wr = csv.DictWriter(output, ['Name','ImageUrl','Height','Weight'], delimiter=',')
    wr.writeheader()

    for i in range(135):
        url = "http://www.host.com/actors/all-actors/name/{}/".format(i)
        response = requests.get(url)
        html = response.content
        soup = BeautifulSoup(html, "html.parser")
        links = soup.find_all("div", { "class" : "card-image" })

        for a in links:
            for url in a.find_all('a'):
                url = "http://www.host.com" + url['href']
                print(url)
                response = requests.get(url)
                html = response.content
                actor_dict = get_actor_dict_from_html(url, html)
                wr.writerow(actor_dict)
    print('All Done!')

if __name__ == "__main__":
    lotta_downloads()

谢谢！

Answer 1

您为什么不尝试使用gevent库？

gevent库的monkey patch使阻止功能变为非阻止功能。

也许请求的wait time太长而太慢。

因此，我认为将请求作为非阻塞函数可以使您的程序运行更快。

在python 2.7.10示例中：

import gevent
from gevent import monkey; monkey.patch_all() # Fix import code
import reqeusts

actor_dict_list = []

def worker(url):
    content = requests.get(url).content
    bs4.BeautifulSoup(content)
    links = soup.find_all('div', {'class': 'card-image'})

    for a in links:
        for url in a.find_all('a'):
            response = requests.get(url) # You can also use gevent spawn function on this line
            ...
            actor_dict_list.append(get_actor_dict_from_html(url, html)) # Because of preventing race condition

output = open("/tmp/export.csv", "w", newline='')
wr = csv.DictWriter(output, ['Name', 'ImageUrl', 'Height', 'Weight'], delimiter=',')
wr.writeheader()

urls = ["http://www.host.com/actors/all-actors/name/{}/".format(i) for i in range(135)]
jobs = [gevent.spawn(worker, url) for url in urls]
gevent.joinall(jobs)
for i in actor_dict_list:
    wr.writerow(actor_dict)

公共gevent文件： doc

聚苯乙烯

如果您有ubuntu OS，则必须安装python-gevent

sudo apt-get install python-gevent