无法使用 concurrent.futures 加速脚本的执行过程

Question

我正在尝试在以下脚本中实现concurrent.futures或类似的东西，以加快执行速度。 但是，当我比较这两个脚本时，我发现速度没有变化。 我应该带来什么可能的改变来实现同样的目标？

原脚本：

import requests
from bs4 import BeautifulSoup


link = 'https://ldc.lloyds.com/market-directory/results'

params = {
    'cobc': '','cob': '','loc': '','ltti': '',
    'bro': '0','cov': '1','man': '0','mem': '0',
    'omc': '0','run': '0','name': '','mode':' cov',
    'c_page': 1 #---------------->unknown number of pages to traverse
}

def get_content(s,link,params):
    while True:
        r = s.get(link,params=params)
        soup = BeautifulSoup(r.text,"lxml")
        if not soup.select(".marketing-directories-results .contact-details > h2"): 
            return

        for item in soup.select(".marketing-directories-results .contact-details > h2"):
            yield item.text

        params['c_page']+=1

if __name__ == '__main__':
    with requests.Session() as s:
        s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
        for item in get_content(s,link,params):
            print(item)

当我在脚本中实现concurrent.futures时，我在执行过程中看不到任何与速度相关的改进：

import requests
from bs4 import BeautifulSoup
import concurrent.futures as futures

link = 'https://ldc.lloyds.com/market-directory/results'

params = {
    'cobc': '','cob': '','loc': '','ltti': '',
    'bro': '0','cov': '1','man': '0','mem': '0',
    'omc': '0','run': '0','name': '','mode':' cov',
    'c_page': 1  #---------------->unknown number of pages to traverse
}

def get_content(s,link,params):

    while True:
        item_list = []
        r = s.get(link,params=params)
        soup = BeautifulSoup(r.text,"lxml")
        if not soup.select(".marketing-directories-results .contact-details > h2"): 
            return

        for item in soup.select(".marketing-directories-results .contact-details > h2"):
            print(item.text)

        params['c_page']+=1
        

if __name__ == '__main__':
    with requests.Session() as s:
        s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'

        with futures.ThreadPoolExecutor(max_workers=5) as executor:
            future_to_url = {executor.submit(get_content, s, url, params): url for url in [link]}
            futures.as_completed(future_to_url)

如何实现concurrent.futures或类似的东西以加快执行速度？

Answer 1

在您的示例中，您将link url 变量作为executor.submit function 的参数中的列表传递。 它将被视为['h','t','t','p',.. so on] 。

我建议这样实现：

import concurrent.futures as futures
from functools import partial

thread_func = partial(get_content,s, url)
with concurrent.futures.ThreadPoolExecutor(max_worker=5) as pool:
    result = pool.map(thread_func, params)

我使用了部分 function 只是为了清楚在 map function 中作为迭代器参数传递的内容。

希望你会发现这很有帮助。