worker 失败时更新全局变量（Python multiprocessing.pool ThreadPool）

Question

I have a Python function that requests data via API and involves a rotating expiring key. The volume of requests necessitates some parallelization of the function. I am doing this with the multiprocessing.pool module ThreadPool. Example code:

import requests
from multiprocessing.pool import ThreadPool
from tqdm import tqdm

# Input is a list-of-dicts results of a previous process.
results = [...]

# Process starts by retrieving an authorization key.
headers = {"authorization": get_new_authorization()}

# api_call() is called on each existing result with the retrieved key.
results = thread(api_call, [(headers, result) for result in results])

# Function calls API with passed headers for given URL and returns dict.
def api_call(headers_plus_result):
    headers, result = headers_plus_result
    r = requests.get(result["url"]), headers=headers)
    return json.loads(r.text)

# Threading function with default num_threads.
def thread(worker, jobs, num_threads=5):
    pool = ThreadPool(num_threads)
    results = list()
    for result in tqdm(pool.imap_unordered(worker, jobs), total=len(jobs)):
        if result:
            results.append(result)
    pool.close()
    pool.join()
    if results:
        return results

# Function to get new authorization key.
def get_new_authorization():
    ...
    return auth_key

I am trying to modify my mapping process so that, when the first worker fails (ie the authorization key expires), all other processes are paused until a new authorization key is retrieved. Then, the processes proceed with the new key.

Should this be inserted into the actual thread() function? If I put an exception in the api_call function itself, I don't see how I can stop the pool manager or update the header being passed to other workers.

Additionally: is using ThreadPool even the best method if I want this kind of flexibility?

Answer 1

A simpler possibility might be to use a multiprocessing.Event and a shared variable. The Event would indicate whether the authentication was legit or not, and the shared variable would contain the authentication.

event = mp.Event()
sharedAuthentication = mp.Array('u', 100) # 100 = max length

So a worker would run:

event.wait();
authentication = sharedAuthentication.value

Your main thread would initially set the authentication with

sharedAuthentication.value = ....
event.set()

and later modify the authentication with

event.clear()
... calculate new authentication
sharedAuthentication.value = .....
event.set()