Unexpected behaviour in python multiprocessing

Question

I'm trying to understand the following odd behavior observed using the python mutiprocessing .

Sample testClass: import os import multiprocessing

class testClass(multiprocessing.Process):    

    def __del__(self):
        print "__del__ PID: %d" % os.getpid()
        print self.a    

    def __init__(self):
        multiprocessing.Process.__init__(self)
        print "__init__ PID: %d" % os.getpid()
        self.a = 0    

    def run(self):
        print "method1 PID: %d" % os.getpid()
        self.a = 1

And a little test program: from testClass import testClass

print "Start"
proc_list = []
proc_list.append(testClass())
proc_list[-1].start()    

proc_list[-1].join()    

print "End"

This produces:

Start
__init__ PID: 89578
method1 PID: 89585
End
__del__ PID: 89578
0

Why it does not print 1 ?

I'm guessing that it's related to the fact that run is actually being executed on a different process as can be seen. If this is the expected behavior how is everyone using multiprocessing where processes have an expensive __init__ as in processes that need to open a database?

And shouldn't this behaviour be better highlighted in multiprocessing documentation?

Answer 1

You can wrap your expensive initialization inside a context manager:

def run(self):
    with expensive_initialization() as initialized_object:
        do_some_logic_here(initialized_object)

You will have a chance to properly initialize your object before calling do_some_logic_here , and to properly release the resources after leaving the context manager's block.

See documentation .

Answer 2

When calling start() , the interpreter forks and creates a child process, which gets a copy of the page tables from the parent. These point to pages which are marked as readonly and only copied when written (COW). The interpreter, when executing run in the child process, accesses the child's copy of the PyObject which represents a . The parent's memory is not touched. The child also gets a copy of the file descriptors table, which means that if a connection is opened by the parent, that file descriptor is inherited by the child. You can see (with strace ) that the child is created via clone without CLONE_FILES :

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f460d8929d0)

Therefore, from clone man page:

If CLONE_FILES is not set, the child process inherits a copy of all file descriptors opened in the calling process at the time of clone(). (The duplicated file descriptors in the child refer to the same open file descriptions (see open(2)) as the corresponding file descriptors in the calling process.) Subsequent operations that open or close file descriptors, or change file descriptor flags, performed by either the calling process or the child process do not affect the other process.