how to kill zombie processes created by multiprocessing module?

Question

I'm very new to multiprocessing module. And I just tried to create the following: I have one process that's job is to get message from RabbitMQ and pass it to internal queue ( multiprocessing.Queue ). Then what I want to do is : spawn a process when new message comes in. It works, but after the job is finished it leaves a zombie process not terminated by it's parent. Here is my code:

Main Process:

 #!/usr/bin/env python

 import multiprocessing
 import logging
 import consumer
 import producer
 import worker
 import time
 import base

 conf = base.get_settings()
 logger = base.logger(identity='launcher')

 request_order_q = multiprocessing.Queue()
 result_order_q = multiprocessing.Queue()

 request_status_q = multiprocessing.Queue()
 result_status_q = multiprocessing.Queue()

 CONSUMER_KEYS = [{'queue':'product.order',
                   'routing_key':'product.order',
                   'internal_q':request_order_q}]
 #                 {'queue':'product.status',
 #                  'routing_key':'product.status',
 #                  'internal_q':request_status_q}]

 def main():
     # Launch consumers
     for key in CONSUMER_KEYS:
         cons = consumer.RabbitConsumer(rabbit_q=key['queue'],
                                        routing_key=key['routing_key'],
                                        internal_q=key['internal_q'])
         cons.start()

     # Check reques_order_q if not empty spaw a process and process message
     while True:
         time.sleep(0.5)
         if not request_order_q.empty():
             handler = worker.Worker(request_order_q.get())
             logger.info('Launching Worker')
             handler.start()

 if __name__ == "__main__":
     main()

And here is my Worker:

 import multiprocessing
 import sys 
 import time
 import base

 conf = base.get_settings()
 logger = base.logger(identity='worker')

 class Worker(multiprocessing.Process):

     def __init__(self, msg):
         super(Worker, self).__init__()
         self.msg = msg 
         self.daemon = True

     def run(self):
         logger.info('%s' % self.msg)
         time.sleep(10)
         sys.exit(1)

So after all the messages gets processed I can see processes with ps aux command. But I would really like them to be terminated once finished. Thanks.

Answer 1

A couple of things:

1. Make sure the parent joins its children, to avoid zombies. See Python Multiprocessing Kill Processes

2. You can check whether a child is still running with the is_alive() member function. See http://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process

Answer 2

Using multiprocessing.active_children is better than Process.join . The function active_children cleans any zombies created since the last call to active_children . The method join awaits the selected process. During that time, other processes can terminate and become zombies, but the parent process will not notice, until the awaited method is joined. To see this in action:

import multiprocessing as mp
import time


def main():
    n = 3
    c = list()
    for i in xrange(n):
        d = dict(i=i)
        p = mp.Process(target=count, kwargs=d)
        p.start()
        c.append(p)
    for p in reversed(c):
        p.join()
        print('joined')


def count(i):
    print('{i} going to sleep'.format(i=i))
    time.sleep(i * 10)
    print('{i} woke up'.format(i=i))


if __name__ == '__main__':
    main()

The above will create 3 processes that terminate 10 second apart each. As the code is, the last process is joined first, so the other two, which terminated earlier, will be zombies for 20 seconds. You can see them with:

ps aux | grep Z

There will be no zombies if the processes are awaited in the sequence that they will terminate. Remove the reversed to see this case. However, in real applications we rarely know the sequence that children will terminate, so using join will result in some zombies.

The alternative active_children does not leave any zombies. In the above example, replace the loop for p in reversed(c): with:

while True:
    time.sleep(1)
    if not mp.active_children():
        break

and see what happens.

Answer 3

Use active_children. multiprocessing.active_children