Share read-only objects between processes

Question

I use a "keep alive" processes model (who communicates with Pipes with main process) in my software, and I'm trying to share read-only objects between them and the main process. An example to show my problem:

from multiprocessing import Process, Pipe#, Manager
from multiprocessing.connection import wait
import os


def start_in_oneshot_processes(obj, nb_process):
    """
    Start nb_process processes to do the job. Then process finish the job they die.
    """
    processes = []
    for i in range(nb_process):
        #  Simple process style
        p = Process(target=oneshot_in_process, args=(obj,))
        p.start()
        processes.append(p)

    for process in processes:
        # Wait all process finish
        process.join()

def oneshot_in_process(obj):
    """
    Main job (don't matter if in oneshot, keeped alive process. It have job to do)
    """
    print('p', obj, os.getpid())


def start_in_keepedalive_processes(obj, nb_process):
    """
    Start nb_process and keep them alive. Send job to them multiple times, then close thems.
    """
    processes = []
    readers_pipes = []
    writers_pipes = []
    for i in range(nb_process):
        # Start process with Pipes for communicate
        local_read_pipe, local_write_pipe = Pipe(duplex=False)
        process_read_pipe, process_write_pipe = Pipe(duplex=False)
        readers_pipes.append(local_read_pipe)
        writers_pipes.append(process_write_pipe)
        p = Process(target=run_keepedalive_process, args=(local_write_pipe, process_read_pipe, obj))
        p.start()
        processes.append(p)
    # Send to process some job to do
    for job in range(3):
        print('send new job to processes:')
        for process_number in range(nb_process):
            # Send data to process
            writers_pipes[process_number].send(obj)
            reader_useds = []
        # Wait response from processes
        while readers_pipes:
            for r in wait(readers_pipes):
                try:
                    r.recv()
                except EOFError:
                    pass
                finally:
                    reader_useds.append(r)
                    readers_pipes.remove(r)
        readers_pipes = reader_useds

    # Kill processes
    for writer_pipe in writers_pipes:
        writer_pipe.send('stop')

def run_keepedalive_process(main_write_pipe, process_read_pipe, obj):
    """
    Procees who don't finish while job to do
    """
    while obj != 'stop':
        oneshot_in_process(obj)
        # Send to main process "I've done my job"
        main_write_pipe.send('job is done')
        # Wait for new job to do (this part can be simplified no ?)
        readers = [process_read_pipe]
        while readers:
            for r in wait(readers):
                try:
                    obj = r.recv()
                except EOFError:
                    pass
                finally:
                    readers.remove(r)


obj = object()
print('m', obj, os.getpid())

print('One shot processes:')
start_in_oneshot_processes(obj, 5)

print('Keeped alive processes:')
start_in_keepedalive_processes(obj, 5)

print('f', obj, os.getpid())

Output is:

➜  sandbox git:(dev/opt) ✗ python3.4 sharedd.py
m <object object at 0xb7266dc8> 3225
One shot processes:
p <object object at 0xb7266dc8> 3227
p <object object at 0xb7266dc8> 3226
p <object object at 0xb7266dc8> 3229
p <object object at 0xb7266dc8> 3228
p <object object at 0xb7266dc8> 3230
Keeped alive processes:
p <object object at 0xb7266dc8> 3231
p <object object at 0xb7266dc8> 3232
send new job to processes:
p <object object at 0xb7266dc8> 3235
p <object object at 0xb7266dc8> 3233
p <object object at 0xb7266dc8> 3234
p <object object at 0xb7266488> 3231
send new job to processes:
p <object object at 0xb7266488> 3232
p <object object at 0xb7266488> 3235
p <object object at 0xb7266488> 3234
p <object object at 0xb7266490> 3231
p <object object at 0xb7266488> 3233
p <object object at 0xb7266490> 3232
p <object object at 0xb7266490> 3235
p <object object at 0xb7266490> 3233
send new job to processes:
p <object object at 0xb7266488> 3232
p <object object at 0xb7266488> 3235
p <object object at 0xb7266490> 3234
p <object object at 0xb7266488> 3231
f <object object at 0xb7266dc8> 3225
p <object object at 0xb7266488> 3233
p <object object at 0xb7266488> 3234

If I create simple processes (start_in_oneshot_processes), obj have the same memory address in subprocess and in main process: 0xb7266dc8 .

But when my process receive objects with a Pipe (start_in_keepedalive_processes), objects memory address are not the same as the main process: example: 0xb7266488 instead of 0xb7266dc8 . These objects are read-only by sub processes. How can I share them between main process and subprocess (and save memory copy time) ?

Answer 1

You can't achieve what you want using Pipe . When data is sent through a pipe to another process, that process has to store a copy of the data in its own address space --- a new object, in Python. Most other methods of shipping data between processes are the same.

It's only coincidence that you saw the same memory address from the start_in_oneshot_process function, likely due to your choice of operating system. In general, two processes won't share the same RAM at all. (Check the Process docs section on Contexts and Start Methods for the differences between Windows spawn and Unix fork .)

If it's really important that two processes can inspect the same chunk of memory, you can try either shared memory or an object manager process . Note that those same docs tell you that this rarely a good idea.