具有不同概率的矢量化 np.random.choice

Question

I've trained a machine learning model using sklearn and want to simulate the result by sampling the predictions according to the predict_proba probabilities. So I want to do something like

samples = np.random.choice(a = possible_outcomes, size = (n_data, n_samples), p = probabilities)

Where probabilities would be is an (n_data, n_possible_outcomes) array

But np.random.choice only allows 1d arrays for the p argument. I've currently gotten around this using a for-loop like the following implementation

sample_outcomes = np.zeros((len(probs), n_samples))
for i in trange(len(probs)):
    sample_outcomes[i, :] = np.random.choice(outcomes, s = n_samples, p=probs[i])

but that's relatively slow. Any suggestions to speed this up would be much appreciated!

Answer 1

Here is an example of what you can do, if I understand your question correctly:

import numpy as np
#create a list of indices
index_list = np.arange(len(possible_outcomes))
# sample indices based on the probabilities
choice = np.random.choice(a = index_list, size = n_samples, p = probabilities)
# get samples based on randomly chosen indices
samples = possible_outcomes[choice]

Answer 2

If I understood correctly you want a vectorize way of applying choice several times and each time with a different probabilities vector. You could implement this by hand as follows:

import numpy as np

# for reproducibility
np.random.seed(42)

# number of samples
k = 5

# possible outcomes
outcomes = np.arange(10)

# generate a random probability matrix for 15 runs
probabilities = np.random.random((15, 10))
probs = probabilities / probabilities.sum(1)[:, None]

# generate the choices by picking those probabilities above a random generated number
# the higher the value in probs the higher the probability to pick it
choices = probs - np.random.random((15, 10))

# to pick the top k using argpartition need to multiply by -1
choices = -1 * choices

# pick the top k values
res = outcomes[np.argpartition(choices, k, axis=1)][:, :k]

# flatten to match the expected output
print(res.flatten())

Output

[1 8 2 5 3 6 4 8 7 0 1 5 9 3 7 1 4 9 0 8 5 0 4 3 6 8 5 1 2 6 5 3 2 0 6 5 4
 2 3 7 7 9 4 6 1 3 6 4 2 1 4 9 3 0 1 6 9 2 3 8 5 4 7 6 1 5 3 8 2 1 1 0 9 7
 4]

In the above example the code sample 5 ( k ) elements from a population of 10 ( outcomes ) 15 times each time with a different probability vector ( probs with a shape of 15 by 10).

Answer 3

I'm making sure I understand you problem correctly. Can you just create samples as an array of size n_data * n_samples and then use the resize method to get it to the right size?

samples = np.random.choice(a = possible_outcomes, size = n_data * n_samples, p = probabilities)
samples.resize((n_data, n_samples))

Answer 4

If you use NumPy's newer interface to random number generation, what you want should be simple: https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.choice.html

See here for an example:

samples = np.random.choice(array_of_samps, n_samples, p=probs)

Note, though, that len(probs) would equal array_of_samps.shape[0] (ie the number of rows in array_of_samps ), not samples.shape[0] . Each row of samples would be a randomly chosen row of array_of_samps .

Judging from the shape of your sample_outcomes array, sample_outcomes is probably samples.T .