如何将特定数量的额外工作人员添加到现有的多处理池中？

Question

在以下情况下，我创建了一个包含两个工作人员的默认池并执行任务。 在任务处理期间，task_queue 会被定期检查，因此它不会超过一定的长度限制并防止上游/下游混乱。 如何动态添加更多工作人员以减少任务队列长度？

import multiprocessing as mp

... code snippet...

def main(poolsize, start_process):

    pool = mp.Pool(processes=poolsize, initializer=start_process)
    done = False

    task_queue = []

    while True:

        ... snippet code : do something ...

        if len(task_queue) >= 10:

            ... code to expand pool goes here...

        if done == True:
            break

    .. do final something ...

if __name__ == '__main__':

#    freeze_support()

    poolsize = 2

    main(poolsize)

Answer 1

要在运行池处理作业期间添加更多工作人员，您可以在 while 循环中添加以下函数：

def repopulate(pool, add_workers):

    current_pool_size = len(pool._pool)         # _.pool gets the current pool size.

    new_pool_size = current_pool_size + add_workers

    pool._processes = new_pool_size

    pool._repopulate_pool()

    return pool

在main()的 while 循环中：

if len(task_queue) >= 10:

    new_workers = 2

    repopulate(poolname, new_workers)

Answer 2

multiprocessing.Pool具有动态大小

基于 ZF007 答案的完整示例

import multiprocessing, time, random

def worker_function(job_id):
    dt = random.randint(1, 10)
    print(f"job {job_id}: sleeping for {dt} seconds")
    time.sleep(dt)
    return job_id * job_id

def get_job_done(job_id):
    return lambda val: print(f"job {job_id}: job done: val={val}")

def grow_pool(pool, new_size, max_size=None):
    new_size = min(new_size, max_size) if max_size else new_size
    if new_size > pool._processes:
        print(f"growing pool from {pool._processes} to {new_size}")
        pool._processes = new_size
        pool._repopulate_pool()

if __name__ == "__main__":

    # start pool
    start_workers = 1 # start N workers before demand
    max_workers = 4 # run N workers on demand
    pool = multiprocessing.Pool(start_workers)

    # add jobs
    num_jobs = 10
    grow_pool(pool, num_jobs, max_workers)
    for job_id in range(0, num_jobs):
        job_done = get_job_done(job_id)
        print(f"job {job_id}: adding job")
        pool.apply_async(worker_function, args=(job_id,), callback=job_done)

    # wait
    pool.close()
    pool.join()