基于频率数据高效拟合 Scipy 中的分布

Question

I have some data that I want to fit to a distribution. The data is given by the frequency. What I mean is, I have every event that I have observed and the number of times that I have observed it. So something like:

data = [(1, 34), (2, 1023), (3, 3243), (4, 879), (5, 202), (6, 10)]

where the first number in each tuple is the event I have observed, and the second number is the total observations for that event.

With Scipy, I can fit (for example) a lognormal distribution using a call to scipy.stats.lognorm.fit. However, this routine expects to see a list of all of the observations, not the frequencies. I can fit the distribution like this:

import scipy
temp_data = []
for x in data:
    temp_data += [x[0]] * x[1]
params = scipy.stats.lognorm.fit(temp_data)

but wow, that seems horribly inefficient.

Is there a to fit a distribution, in Scipy or other similar tool, based upon the frequencies? If not, is there a better way to fit the distribution without having to create a potentially giant list of values?

Answer 1

Unfortunately, looking at the source , it seems like the 'materialized' aspect of the data is hardcoded. The function's not that complicated, though, so you could make your own version. TBH if your total N is still manageable I'd probably just do data = np.array(data); expanded_data = np.repeat(data[:,0], data[:,1]) data = np.array(data); expanded_data = np.repeat(data[:,0], data[:,1]) despite the inefficiency, because life is short.

Another alternative would be to use pomegranate , which supports passing weights:

import numpy as np
import scipy.stats
import matplotlib.pyplot as plt
import pomegranate as pg

data = [(1, 34), (2, 1023), (3, 3243), (4, 879), (5, 202), (6, 10)]

data = np.array(data)
expanded = np.repeat(data[:,0], data[:,1].astype(int))

scipy_shape, _, scipy_scale = scipy_params = scipy.stats.lognorm.fit(expanded, floc=0)
scipy_sigma, scipy_mu = scipy_shape, np.log(scipy_scale)

pg_dist = pg.LogNormalDistribution(0, 1)
pg_dist.fit(data[:,0], weights=data[:,1])
pg_mu, pg_sigma = pg_dist.parameters

fig = plt.figure()
ax = fig.add_subplot(111)

x = np.linspace(0.1, 10, 100)
ax.plot(data[:,0], data[:, 1] / data[:,1].sum(), label="freq")
ax.plot(x, scipy.stats.lognorm(*scipy_params).pdf(x),
        label=r"scipy: $\mu$ {:1.3f} $\sigma$ {:1.3f}".format(scipy_mu, scipy_sigma), alpha=0.5)
ax.plot(x, pg_dist.probability(x),
        label=r"pomegranate: $\mu$ {:1.3f} $\sigma$ {:1.3f}".format(pg_mu, pg_sigma), linestyle='--', alpha=0.5)
ax.legend(loc='upper right')
fig.savefig("compare.png")

gives me

Answer 2

You can draw a random sample according to you frequency distribution, and fit that:

import scipy
import numpy as np

data = np.array(
    [(1, 34), (2, 1023), (3, 3243), (4, 879), (5, 202), (6, 10)], 
    dtype=float,
)
values = data[0]
weights = data[1]
seed = 87

gen = np.random.default_rng(seed)
sample = gen.choices(
    values, size=500, p=weights/np.sum(weights))

params = scipy.stats.lognorm.fit(values)