python 多处理子进程未正常退出

Question

I've being using python multiprocessing for some task handling. The dev enviroment is Windows Server 2016 and python 3.7.0.
Sometimes there were child processes that stayed in the task list. But actually, they seemed to be completed(data were writen into database). The impact is that the logging stucked there, being unable to append latest logs.

Here is the code. Main function starts a listener process and several worker processes:

queue = multiprocessing.Queue(-1)
listener = multiprocessing.Process(target=listener_process, args=(queue, listener_configurer))
listener.start()

...

workers = []
for loop:
    worker = multiprocessing.Process(target=process_start, args=(queue, worker_configurer, plist))
    workers.append(worker)
    worker.start()
for w in workers:
    w.join()

...

queue.put_nowait(None)
listener.join()

The listener process ends when it gets None, thus resulting the whole task to end.

def listener_process(queue, configurer):
    configurer()
    while True:
        try:
            record = queue.get()
            if record is None:
                break
            if type(record) is not int:
                Logger = logging.getLogger(record.name)
                Logger.handle(record)
        except Exception as e:
            Logger.error(str(e), exc_info=True)

Task is scheduled to run by windows task scheduler.
Any idea why some multiprocessing processes were 'stuck' there?
It's being bothering me for some time. Thanks in advance.

Answer 1

Can I say for sure what is your problem? No. Can I say for sure you are doing something that can lead to a deadlock? Yes.

If you read the documentation carefully on multiprocessing.Queue , you will see the following warning:

Warning: As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread ), then that process will not terminate until all buffered items have been flushed to the pipe.

This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.

Note that a queue created using a manager does not have this issue. See Programming guidelines.

This means that to be completely safe, you must join the listener process (which is issuing gets from the queue) first before joining the workers processes (which are issuing puts to the queue) to ensure that all the messages put to the queue have been read off the queue before you attempt to join the tasks that have done the puts to the queue.

But then how will the listener process know when to terminate if currently it is looking for the main process to write a None sentinel message to the queue signifying that it is quitting time but in the new design the main process must first wait for the listener to terminate before it waits for the workers to terminate? Presumably you have control over the source of the process_start function that implements the producer of messages that are written to the queue and presumably something triggers its decision to terminate. When these processes terminate it is they that must each write a None sentinel message to the queue signifying that they will not be producing any more messages. Then funtion listener_process must be passed an additional argument, ie the number of message producers so that it knows how many of these sentinels it should expect to see. Unfortunately, I can't discern from what you have coded, ie for loop: , what that number of processes is and it appears that you are instantiating each process with identical arguments. But for the sake of clarity I will modify your code to something that is more explicit:

queue = multiprocessing.Queue(-1)
listener = multiprocessing.Process(target=listener_process, args=(queue, listener_configurer, len(plist)))
listener.start()

...

workers = []
# There will be len(plist) producer of messages:
for p in plist:
    worker = multiprocessing.Process(target=process_start, args=(queue, worker_configurer, p))
    workers.append(worker)
    worker.start()
listener.join() # join the listener first
for w in workers:
    w.join()


....


def listener_process(queue, configurer, n_producers):
    configurer()
    sentinel_count = 0
    while True:
        try:
            record = queue.get()
            if record is None:
                sentinel_count += 1
                if sentinel_count == n_producers:
                    break # we are done
                continue
            if type(record) is not int:
                Logger = logging.getLogger(record.name)
                Logger.handle(record)
        except Exception as e:
            Logger.error(str(e), exc_info=True)