Why python multiprocessing always generate same random values

Question

I am using pool.map to process my function. There is a random value added in the function. But I find the result from each process is the same. How can I generate different random values for each process. This is an example:

import numpy as np
from numpy.random import randn
def testfun(X):
    scipy.random.seed
    y = randn()
    return y

from multiprocessing import Pool


pool = mp.Pool(processes = 8)

result = pool.map(testfun,np.arange(8))

I want to have 8 different values.

Answer 1

scipy.random.seed just references the function. You need to actually call it with scipy.random.seed() .

Answer 2

You need to provide different seed values. The value X you get from the range will do it. I was not able to execute your code, but I created a simplified version of it:

from multiprocessing import Pool
import random


def testfun(seed_: int):
    random.seed(seed_)
    y = random.gauss(0, 1)
    return y


if __name__ == "__main__":
    pool = Pool(8)
    result = pool.map(testfun, range(8))
    print(result)

It is also better to put the pool into a with context manager.

Answer 3

What you need to do is to seed the random number generator once for each process in the multiprocessing pool (rather than for each call to rand ) and the way to do that is with a pool initializer, ie specifying the initializer argument on the multiprocessing.Pool constructor. This function you specify will be called once for each process in the pool before any tasks are executed and can be used to perform any one-time initialization such as setting global variables or, in this case, seeding the random number generator for this process.

import numpy as np
from numpy.random import randn, seed


def init_pool_processes():
    seed()

def testfun(X):
    y = randn()
    return y

# Required by Windows:
if __name__ == '__main__':
    from multiprocessing import Pool

    pool = Pool(processes=8, initializer=init_pool_processes)

    result = pool.map(testfun, np.arange(8))
    print(result)

Prints:

[-0.01738709180801345, -0.6941424935875462, 0.41955492420787543, -0.890711442154167, -0.6894630549510319, 1.1549486347982545, -0.27329303494286733, 0.16447656347746123]