Update different global variable using Multiprocessing in Python

Question

I have a script that all my workers updates different dictionary in a Manager list. My question is: when one worker is writing the list, will the others wait for it, or all worker can update the list at the same time?

Here is my code:

from multiprocessing import Process, Manager

def worker(x, i, *args):
    sub_l = x[i]
    sub_l[i] = i
    x[i] = sub_l


if __name__ == '__main__':
    manager = Manager()
    num = 4
    x = manager.list([{}]*num)
    p = []
    for i in range(num):
        p.append(Process(target=worker, args=(x, i)))
        p[i].start()

    for i in range(5):
        p[i].join()

    print x

I only need all my workers to run separately, and update different global variables. I kind of think using manager.list is an over kill, but not sure if there is other way to do this.

Answer 1

The Manager server that provides access to the Manager.list doesn't do any synchronization when you try to access objects its managing. You can basically think of it exactly the same way you would if you were just dealing with threads and an ordinary global variable; because of the GIL, two threads aren't going to be able to actually step on top of each other while doing an atomic bytecode operation, but doing something like incrementing a variable, which requires multiple bytecode operations, need to be protected by a lock.

In your case, you'd only be in trouble if multiple workers did certain operations on the same sub-list in parallel. If two workers ran this at the same time:

sub_l = x[i]
sub_l[i] = sub_l[i] + <something unique to the worker>
x[i] = sub_l

With the same i, they could end up stomping on each other; both would store a copy of the same sub_l , and both would increment sub_l[i] , and then both would update x[i] , but the second one to update x would overwrite the changes done by the first one.

As long as you don't try to do those kind of things in parallel across workers, you should be fine.