在 python/numpy 中生成 ~10^9 泊松随机数的最快方法

Question

I would like to find the fastest way to generate ~10^9 poisson random numbers in python/numpy—for instance, say I have a mean Poisson parameter (calculated elsewhere) of shape (1000, 2000), and I need 500 independent samples. This is a bottleneck in my code, taking several minutes to complete. I have tried three methods, but am looking for something faster:

import numpy as np

# example parameters
nsamples = 500
nmeas = 2000
ninputs = 1000
lambdax = np.ones([ninputs, nmeas]) * 20

# numpy, one big array
sample0 = np.random.poisson(lam=lambdax, size=(nsamples, ninputs, nmeas))

# numpy, current version where other code happens in the loop
sample1 = np.zeros([nsamples, ninputs, nmeas])
for i in range(nsamples):
    sample1[i, :, :] = np.random.poisson(lam=lambdax)

# scipy
from scipy.stats import poisson
sample2 = poisson.rvs(lambdax, size=(nsamples, ninputs, nmeas))

Results:

sample0: 1 m 16 s
sample1: 1 m 20 s
sample2: 1 m 50 s

Not shown here, I am also parallelizing the independent samples via multiprocessing , but the calculations are still pretty expensive for such large parameters. Is there a better way?

Answer 1

I have been in your shoes and here are my suggestions:

• For large mean values, poisson works similar to uniform. check out this post (and probably more if you search).
• ~1m runtime seems reasonable to generate such a large number of random numbers. I don't think you can top sample0 method by much via just coding. Now depending on what you want to do with random numbers,
  • if your issue is rerunning program multiple times, try saving sample0 into a file and reloading it in the next runs.
  • if not, I suggest creating lower number of randoms and reuse them. A lot of those random numbers in sample0 will be repeated in your sample, depending on your mean value. You might want to create smaller sample size and randomly choose from them. for example I would chose a random number from sample0 and reuse it for eg 100 times (since that number would appear in sample0 over 100 times anyways).

If you provide more information on what you intend to do with random numbers, we might be able to help more. Otherwise, coding-wise I am not sure if you can do much further.