如何使用 Python 多处理池永远消耗队列中的项目

Question

I'm trying to create a worker that listens to http requests and adds jobs IDs to a queue. I'm using Python's built-in multiprocessing module for that.

I need a Pool with a few processes that will process the job from queue and respawn. Processes have to restart, bacause for some cases job processing can cause memory leak. Pool should run forever as the items will be added to the queue dynamically.

The problem is that my pool does not respawn workers after they complete.

How can I use pool to achieve this? I want it to run forever, consume item from queue and respawn child after every task.

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from multiprocessing import Pool, SimpleQueue, current_process

queue = SimpleQueue()

def do_something(q):
    worker_id = current_process().pid
    print(f"Worker {worker_id} spawned")
    item_id = q.get()
    print(f"Worker {worker_id} received id: {item_id}")
    # long_term_operation_that_leaks_memory(item_id)
    # print(f"Worker {worker_id} completed id: {item_id}")

def main():
    with Pool(
        processes=2, initializer=do_something, initargs=(queue,), maxtasksperchild=1
    ):
        queue.put("a")
        queue.put("b")
        queue.put("c")
        server_address = ("", 8000)
        httpd = ThreadingHTTPServer(server_address, BaseHTTPRequestHandler)
        try:
            httpd.serve_forever()
        except (KeyboardInterrupt, SystemExit):
            pass

if __name__ == "__main__":
    main()

I tried with initializer and maxtasksperchild but it does not work.

I know I can add new processes to a pool using map, but I don't have a map of an infinite possible tasks from the future. I think initializer should be responsible for all new tasks. But I don't know how to force it to run forever and respawn.

In my code example "c" item is never processed. Therefore if I add http logic to put more items it will not work either. Adding http logic to this code is not necessary part of my question, but any tips will be welcomed.

Thanks!

Edit:

The reason I decided to use Pool in this case, is that official documentation says:

Worker processes within a Pool typically live for the complete duration of the Pool's work queue. A frequent pattern found in other systems (such as Apache, mod_wsgi, etc) to free resources held by workers is to allow a worker within a pool to complete only a set amount of work before being exiting, being cleaned up and a new process spawned to replace the old one. The maxtasksperchild argument to the Pool exposes this ability to the end user.

My goals:

• Items will be added dynamically to the queue by http requests
• Pool will live forever
• Worker process will perform only one task from queue and will be respawned

Why I used only 2 processes?

Processes number will not be infinite and it is easy to test my example with 2 processes rather that 5 or 10.

Why I put 3 items manually? It is for example purpose, in real solution all items will be added dynamically, so there is no way to loop over them or to use map on them.

Answer 1

What you are doing with your pool initializer is most unusual. Such an initializer is run for each pool process and is used to initialize that process (for example, setting global variables) so that it is able to run tasks that are submitted . A multiprocessing pool implements a hidden task queue for holding submitted tasks waiting to be processed by an available pool process. Your initializer code is only capable of executing a single quasi-task (I reserve the term task for work submitted to the processing pool in the "normal" way) and then it returns. That is, you are putting 3 items on the queue yet you only have 2 pool processes getting from the queue a single item, processing it and then returning. This does not make any sense to me.

Your code doesn't show the relationship between your HTTP server and running tasks in your multiprocessing pool and I will not guess what that may be. So I will only show the more standard way of using a pool. I have removed the maxtasksperchild argument because it is only relevant when your pool is executing "normal" tasks that are added to the task queue, for example, using the apply_async or map methods. Thus it was not accomplishing anything in your code.

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from multiprocessing import Pool, current_process


def do_something(item_id):
    worker_id = current_process().pid
    print(f"Worker {worker_id} received id: {item_id}")
    # long_term_operation_that_leaks_memory(item_id)
    # print(f"Worker {worker_id} completed id: {item_id}")

def main():
    # Why only 2 processes in the pool?:
    pool = Pool(processes=2)
    pool.apply_async(do_something, args=('a',))
    pool.apply_async(do_something, args=('b',))
    pool.apply_async(do_something, args=('c',))
    server_address = ("", 8000)
    httpd = ThreadingHTTPServer(server_address, BaseHTTPRequestHandler)
    try:
        httpd.serve_forever()
    except (KeyboardInterrupt, SystemExit):
        pass
    # Wait for submitted tasks to complete:
    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

Prints:

Worker 15560 received id: a
Worker 8132 received id: b
Worker 15560 received id: c

Answer 2

It seems to me like maybe you don't really need a pool here, and can maybe just create a new Process for each task. If you want to limit how many tasks exist at once, you can use a Semaphore to limit process creation, and release that semaphore just before each task completes:

from multiprocessing import Process, BoundedSemaphore
from time import sleep

def do_work(A, B):
    sleep(.4)
    print(A, B)

def worker(sema, *args):
    try:
        do_work(*args)
    finally:
        sema.release() #allow a new process to be started now that this one is exiting

def main():
    tasks = zip(range(65,91), bytes(range(65,91)).decode())
    sema = BoundedSemaphore(4) #only every 4 workers at a time
    procs = []
    for arglist in tasks:
        sema.acquire() #wait to start until another process is finished
        procs.append(Process(target=worker, args=(sema, *arglist)))
        procs[-1].start()

        #cleanup completed processes
        while not procs[0].is_alive():
            procs.pop(0)
    for p in procs:
        p.join() #wait for any remaining tasks
    print("done")

if __name__ == "__main__":
    main()