如何提供 multiprocessing.process 唯一变量

Question

我有一个包含 ID 号的列表，我想在 API 调用中为每个多处理器实现每个唯一的 ID 号，同时运行相同的相应功能，对每个处理器实现相同的条件语句等。我试图理解它，但那里网上关于这个程序的不多。

我想使用一个 for 循环，但我不希望每个运行这个 for 循环的处理器都拾取列表中的每个项目。 我只需要将每个项目关联到每个处理器。

我在想这样的事情：

from multiprocessing import process 
import requests, json

ID_NUMBERS = ["ID 1", "ID 2", "ID 3".... ETC]
BASE_URL = "www.api.com"
KEY = {"KEY": "12345"}

a = 0

for x in ID_NUMBERS:
    def[a]():
        while Active_live_data == True:
            # continuously loops over, requesting data from the website
            unique_api_call = "{}/livedata[{}]".format(BASE_URL, x)
            request_it = requests.get(unique_api_call, headers=KEY)
            show_it = (json.loads(request_it.content))

            #some extra conditional code...

        a += 1

processes = []
b = 0

for _ in range(len(ID_NUMBERS))
    p = multiprocessing.Process(target = b)
    p.start()
    processes.append(p)    
    b += 1

任何帮助将不胜感激！

最亲切的问候，

安德鲁

Answer 1

您可以使用map function：

import multiprocessing as mp
num_cores = mp.cpu_count()
pool = mp.Pool(processes=num_cores)
results = pool.map(your_function, list_of_IDs)

这将执行 function your_function ，每次使用列表list_of_IDs中的不同项目，并且your_function返回的值将存储在值列表（ results ）中。

Answer 2

与@AlessiaM 相同的方法，但在concurrent.futures模块中使用高级 api。

import concurrent.futures as mp
import requests, json

BASE_URL = ''
KEY = {"KEY": "12345"}
ID_NUMBERS = ["ID 1", "ID 2", "ID 3"]

def job(id):
    unique_api_call = "{}/livedata[{}]".format(BASE_URL, id)
    request_it = requests.get(unique_api_call, headers=KEY)
    show_it = (json.loads(request_it.content))
    return show_it

# Default to as many workers as there are processors,
# But since your job is IO bound (vs CPU bound), 
# you could increase this to an even bigger figure by giving the `max_workers` parameter
with mp.ProcessPoolExecutor() as pool:
    results = pool.map(job,ID_NUMBERS)

# Process results here