Multiprocessing process queues for multiple types of processes

Question

I am relatively new to Python, and I am trying to create a queue which holds a number of different processes. There are a total of 3 processes called Process1, Process2, and Process3. When Process1 completes its execution, I want a new process Process2 to be added to the queue. When Process2 completes its execution, I want a new process Process3 to be added to the queue.

The reason why I am wanting to use a queue is because if Process2 fails, I want to move this task to the back of the queue so that it can be executed at a later time.

Here is my current implementation:

from multiprocessing import Process, Queue
import time

class Process1(Process):
    def __init__(self, queue):
            super(Process1, self).__init__()
            self.queue = queue

    def run(self):
            print 'I am Process 1'
            time.sleep(1)
            print 'Done process 1'
            p2 = Process2(self.queue)
            self.queue.put(p2)
            p2.start()
            p2.join()

class Process2(Process):
    def __init__(self, queue):
            super(Process2, self).__init__()
            self.queue = queue

    def run(self):
            print 'I am Process 2'
            time.sleep(2)
            print 'Done process 2'
            p3 = Process3(self.queue)
            self.queue.put(p3)
            p3.start()
            p3.join()

class Process3(Process):
    def __init__(self, queue):
            super(Process3, self).__init__()
            self.queue = queue

    def run(self):
            print 'I am Process 3'
            time.sleep(3)
            print 'Done process 3'


if __name__ == '__main__':
    queue = Queue()

    p1 = Process1(queue)
    p1.start()
    p1.join()

When this code is executed, I am getting this error:

RuntimeError: Queue objects should only be shared between processes through inheritance.

I am really having trouble understanding Python Processes and Queues. I have tried reading Python's documentation, but I am having trouble finding a solution for what I am wanting to do. If anyone could provide me with background reading that would be awesome. Thanks!

Answer 1

Multiprocessing queues are to be used by a processes immediate children only. By having Process1 create Process2 and Process2 create Process3, you are trying to hand the queue to the process grandchildren and great-grandchildren. You need to have your top level process (the code in main ) decide when children need to be created and it should also hold the rultes for dealing with failed children and requeuing. Generally, the top level process will make decisions based on result objects when process work is done.