Shared array of objects using multiprocessing in python

Question

I am trying to create an array of shared objects (instances of a class) between different processes in a python program. Each of these objects would be modified in the program. For this purpose I am using multiprocessing as follows:

import multiprocessing
import numpy as np
sh = multiprocessing.RawArray (ctypes.py_object, 10)
f = np.frombuffer(sh, dtype=object)

and the error I get is:

Traceback (most recent call last):
  File "<pyshell#14>", line 1, in <module>
    f = np.frombuffer(sh, dtype=object)
ValueError: cannot create an OBJECT array from memory buffer

Now my question is, first of all, is this generally the right way to tackle this problem, and second, what is my mistake in the above code? Thank you in advance.

Answer 1

ctypes.py_object represents a PyObject * in C. It is a pointer to a struct that represents a Python object, resides in the private memory of the process, and contains more pointers. Another process can't access it; trying to share a pointer between processes is not useful.

See also this answer by Alex Martelli .

Answer 2

You might want to use a multiprocessing.Queue , in which you can just dump objects without worrying about types. It's also thread-safe and process-safe.

Here's a simple example of a Queue used to facilitate the Producer-Consumer problem ( original source , Pizzas were a little bonus by me).

from multiprocessing import Process, Queue

class Pizza(object):
    def __init__(self, pizza_num):
        self.pizza_num = pizza_num
        self.num_slices = 8

sentinel = "NO PIZZA"

def producer(initial_num_pizzas, total_num_pizzas, q):
    """Cooks Pizzas to be consumed and waits for the consumer to finish eating."""
    print("Producer: I am cooking %s Pizzas and putting them on the Queue!"%(total_num_pizzas-initial_num_pizzas))
    for i in range(q.qsize(), total_num_pizzas):
        print("Producer: Behold, for I have cooked Pizza no. %s"%i)
        q.put(Pizza(i))
    q.put(sentinel)

def consumer(q):
    """Consumes some Pizza. In this case, all it does is set the number of slices to 0."""
    while True:
        pizza = q.get()
        pizza.num_slices = 0
        if pizza == sentinel:
            break
        print("Comsumer: Pizza no. %s was found! It has %s slices, yum!"%(pizza.pizza_num, pizza.num_slices))

if __name__ == '__main__':
    q = Queue()
    total_num_pizzas = 10
    initial_num_pizzas = 4
    ## Let's add some Pizzas beforehand:
    for i in range(0, initial_num_pizzas):
        q.put(Pizza(i))
    print("Main: I have precooked %s Pizzas."%q.qsize())

    producer_proc = Process(target=producer, args=(initial_num_pizzas, total_num_pizzas, q))
    consumer_proc = Process(target=consumer, args=(q,))
    producer_proc.start()
    consumer_proc.start()

    q.close()  ## Shop is closed, no more Pizzas will be added to Queue!
    q.join_thread()

    producer_proc.join()
    consumer_proc.join()

Below is an example output. If you run it, the Producer and Consumer print statements may be interleaved differently, because of non-deterministic execution of parallel processes.

Main: I have precooked 4 Pizzas.
Producer: I am cooking 6 Pizzas and putting them on the Queue!
Producer: Behold, for I have cooked Pizza no. 4
Producer: Behold, for I have cooked Pizza no. 5
Producer: Behold, for I have cooked Pizza no. 6
Producer: Behold, for I have cooked Pizza no. 7
Comsumer: Pizza no. 0 was found! It has 8 slices, yum!
Comsumer: Pizza no. 1 was found! It has 8 slices, yum!
Producer: Behold, for I have cooked Pizza no. 8
Comsumer: Pizza no. 2 was found! It has 8 slices, yum!
Producer: Behold, for I have cooked Pizza no. 9
Comsumer: Pizza no. 3 was found! It has 8 slices, yum!
Comsumer: Pizza no. 4 was found! It has 8 slices, yum!
Comsumer: Pizza no. 5 was found! It has 8 slices, yum!
Comsumer: Pizza no. 6 was found! It has 8 slices, yum!
Comsumer: Pizza no. 7 was found! It has 8 slices, yum!
Comsumer: Pizza no. 8 was found! It has 8 slices, yum!
Comsumer: Pizza no. 9 was found! It has 8 slices, yum!

Note that you should use Sentinels to mark the end of your Queue . I've used "NO PIZZA" here, but they can be anything at all.