使用 Joblib 的随机数生成器

Question

I need to generate random number in a function which is paralleled using Joblib. However, the random number generated from the cores are exactly the same.

Currently I solved the problem by assigning random seeds for different cores. Is there any simple way to solve this problem?

Answer 1

This is expected, although unfortunate.

The reason is that joblib (based on the standard multiprocessing Python tool) relies on forking under Unix. Forking creates the exact same processes and thus the same pseudo-random number generation.

The right way to solve this problem is to pass to the function that you are calling in parallel a seed for each call, eg a randomly-generated integer. That seed is then used inside the function to seed the local random number generation.