如何在 Python 3.8 的多处理中共享字符串数组？

Question

I am experimenting with Python's (v. 3.8) multiprocessing library, for developing a bigger program, and trying to share an multiprocessing.Array of strings between multiple processes such that this Array can be updated and read by each process with the same data. I tried c_char_p and it says to use byte strings. However, for this code:

from multiprocessing import Process, Array
from ctypes import c_char_p

def show(a):
    print("This ran")
    print("a: ", a[:])

if __name__ == "__main__":
    array = Array(c_char_p, 1)
    array[0] = b'Hello World'
    print(array[:])
    p = Process(target=show, args=(array,))
    p.start()
    p.join()

the output varies from this:

[b'Hello World']
This ran
a:  [b'c']

to this:

[b'Hello World']
This ran

but I expect:

['Hello World']
This ran
['Hello World']

I guess an obvious solution would be to share a common file between each process but I expect to use multiple arrays and that can get a bit tedious. I was wondering what be the best current solution to this in Python 3.8.

For this code:

from multiprocessing import Process, Array
from ctypes import c_char_p

def show(a):
    print("This ran")

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in a]
    print(arr)

if __name__ == "__main__":
    array = Array(c_char_p, 1)
    message = b"Hello, world"
    array[0] = message

    p = Process(target=show, args=(array,))
    p.start()
    p.join()

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in array]
    print(arr)

it never ends (ie never gets to the print statement after the process starts), the output is:

This ran

I am not sure why.

Answer 1

The values are stored in array as the encoded strings(byte values) , so when you want the string values you have to decode them.

Try this:

from multiprocessing import Process, Array
from ctypes import c_char_p

def show(a):
    print("This ran")

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in a]
    print(arr)

if __name__ == "__main__":
    array = Array(c_char_p, 1)
    array[0] = b"Hello, world"

    # Decode the encoded values
    arr = [s.decode("utf-8") for s in array]
    print(arr)

    p = Process(target=show, args=(array,))
    p.start()
    p.join()

Output:

['Hello, world']
This ran
['Hello, world']

Answer 2

I don't think it is supposed to work in general.

Basically, c_char_p is a pointer, a number that references a memory location in a process. And you cannot expect pointers to be portable between processes.

Since Python Process -es inherit from their parent processes it might work for strings that were created before the child process was started.

But you cannot expect to create a new string in the child process, add a pointer to the Array and expect it to be valid in the parent process. (Nor the other way around.)

Using a Pipe or a Queue is probably a better way to exchange string data.

Edit 1:

Use a Pipe to send mutations of a list of strings, eg as a tuple (3, "new value") . This would mean that the item 3 in the list now becomes "new value" . Using a duplex Pipe means that both ends of the pipe can read and apply every message that comes in.

Edit 2 :

Another option would be to use amultiprocessing.Manager . This will allow you to share list s and dict s.