共享内存和python中多处理的连接问题

Question

I have written some codes as below, through which I wanted to test some specific problems with multiprocessing:

import multiprocessing as mp
import sys
z = 10
file = open("test_file")
file2 = open("test_multiprocess", "w")
arr = []

def func(obj, idx):
    print("pid[%d] [%s]" % (idx,str(id(obj))))
    if idx == 1:
        obj += 3
    elif idx == 2:
        obj = open("test_multiprocess")
    elif idx == 3:
        obj = open("test_multiprocess_%d" % idx, "w")
    elif idx == 4:
        obj.append(idx)
    print("pid[%d] after changing [%s]" % (idx, str(id(obj))))
    sys.stdout.flush()

if __name__ == "__main__":
    data = {1:z, 2:file, 3:file2, 4:arr}
    p = []
    print("original id is [%s] [%s] [%s] [%s]" % (str(id(data[1])), str(id(data[2])), str(id(data[3])), str(id(data[4]))))
    print("==============================================================")
    for i in range(1, 5):
        p.append(mp.Process(target=func, args=(data[i], i)))
        p[len(p)-1].start()

    for i in range(i, len(p)):
        p[i].join()

    sys.stdout.flush()
    print("==============================================================")
    print("after process id is [%s] [%s] [%s] [%s]" % (str(id(data[1])), str(id(data[2])), str(id(data[3])), str(id(data[4]))))

However, when I ran this file, I found some weird phenomena. One particular output is as below:

original id is [6330144] [140069930330512] [140069930330992] [140069873535384]
==============================================================
pid[1] [6330144]
pid[1] after changing [6330072]
pid[2] [140069930330512]
pid[2] after changing [140069864561880]
pid[3] [140069930330992]
pid[3] after changing [140069864561880]
==============================================================
after process id is [6330144] [140069930330512] [140069930330992] [140069873535384]
pid[4] [140069873535384]
pid[4] after changing [140069873535384]

First of all, when passing data[i] to subprocess, id(data[i]) does not change at all, but AFAIK python fork() is copy-on-accessing because of ref-count changing. Secondly, suppose it is copy-on-write in python, when obj is modified in subprocess, int/File type object does change its id but this is not true with type list as we can see its id never changes even compared with the original id. Last but not least, I use join to wait for all subprocesses to complete, but output of parent process seems to always be messed up with subprocess ones, why? Thanks if any one could explain these for me.

Answer 1

None of this has anything to do with multiprocessing. You can run the exact same test just calling the same function in-process and you'll get the same results.

---

First of all, when passing data[i] to subprocess, id(data[i]) does not change at all

That's because you don't change data[i] anywhere.

When you pass data[i] as an argument, the obj parameter doesn't become a reference to the variable data[i] , it becomes a reference to the same value that's in that variable.

When you later do obj = … , that doesn't affect the value in any way; it just makes obj refer to a different value instead.

If you want to mutate data , you have to pass data itself (and, presumably, i ); then the function can do data[i] = .

---

Secondly, suppose it is copy-on-write in python, when obj is modified in subprocess, int/File type object does change its id but this is not true with type list

Nope. The difference has nothing to do with the types, or with copy-on-write. obj = … never affects the original object. It doesn't matter whether it's an int or a list .

The reason you see different behavior isn't that the type is different, it's that you have different code. You don't do obj = … , you do obj.append(…) . That's a method on the object, which mutates it in-place.

(If you're curious about obj += 3 , that one is a bit trickier. Augmented assignment may mutate the value in-place as well as assigning a new value to the variable, or it may just assign a new value to the variable. It's up to the value's type. Generally, mutable types like list will mutate in-place and assign self back to the variable; immutable types like int will of course never mutate in-place, they'll always assign a new value.)

---

My guess is that you're coming from a language like C++, where variables are actually memory locations where the values live, assignment is a mutating method (which normally copies values), and explicit references are references to variables, not to values. If so, it may help to think of every variable in Python a std::shared_ptr<boost::any> , not a boost::any& .

---

Last but not least, I use join to wait for all subprocesses to complete, but output of parent process seems to always be messed up with subprocess ones, why?

Because you don't actually wait for all subprocesses to complete. This line:

for i in range(i, len(p)):

… iterates over range(5, 4) , which is empty, so you don't join anything.

This is one of the many reasons it's better to iterate directly over collections:

for proc in p:
    proc.join()

If you do it that way, there's no place to insert a hard-to-debug counting error.

---

Meanwhile, even though it turns out to be completely irrelevant to your actual code, if you're interested in what does and doesn't get inherited and how, read about start methods and the related programming recommendations . But briefly: if you're using the fork method (the default on non-Windows systems), your globals are shared directly; if you're using spawn (the default on Windows), they're re-constructed from the source. But if you really need to share variables (which you really don't want to do anyway in most cases), you should almost never rely on the fork behavior, even if you're only going to run on Unix; use explicit shared memory . (Also, if you really need to share anything that can be mutated by more than one process, you really need a Lock or other synchronization object.)