Get Data from Other Processes using Multiprocessing

Question

(Language is Python 3)

I am writing a program with the module multiprocessing and using Pool . I need some variable that is shared between all of the processes. The parent process will initialize this variable and pass it as an argument to p.map() . I want the child processes to change this variable. The intent of this is because the first part of the child processes' work should be done in parallel (computational work that doesn't need any other processes' data). But, the second part of the processes' work needs to be done in order, one process after another, because they are writing to a file and the contents of that file should be in order. I want each process to wait until the others are done before moving on. I will record the "progress" of the entire program with the variable, eg when the first process is done writing to the file, it will increment the variable by one. I want this to be a signal to the next process in line to begin writing to the file. But I need some sort of waituntil() function to make the processes wait until the Value variable indicates that it is their "turn" to write to the file.

Here are my two problems:

1. I need a variable that the child processes can edit, and the child processes can actually get the value of that variable. What type of variable should I use? Should I use Value , Manager , or something else?

2. I need the processes to wait until the variable described above equals to a certain value, signaling that it is their turn to write to the file. Is there any sort of waituntil() function that I can use?

Answer 1

What you are looking for is called Synchronization . There are multitudes of different synchronization primitives to choose from.

You should never attempt to write synchronization primitives on your own, as it is non-trivial to do correctly!

In your case either an Event or a Condition might be suitable.