来自不同形状的NumPy数组集合的组合均值和标准差

Question

Let's say I have Numpy arrays with shapes

(682, 89, 138)
(2668, 76, 89)
(491, 62, 48)

How should I calculate the mean and standard deviation of all three arrays combined? If they were the same shapes, I could use np.stack() and then get the mean and std of the resulting array.

Is it possible to do this with different sized dimensions? Or would I have to reshape before getting the mean and std?

Answer 1

We could use the formula of standard deviation and mean to compute those two scalar values for all input arrays without concatenating/stacking (that could be costly specially on large NumPy arrays). Let's do it in steps - mean and then standard deviation, as it seems we could use mean in std computations.

Getting the combined mean value :

So, we will start with the mean/averaging. For this, we would get the summation scalar for each array. Then, get the total summation and finally divide by the number of elements in all arrays.

Getting the combined standard deviation value :

For standard deviation, we have the formula as :

So, we will use the combined mean value obtained from previous step, use the std formula to get the squared differentiation, divide by the total number of elements across all arrays and then apply square root.

Implementation

Let's say the input arrays are a and b , we would have one solution, like so -

N = float(a.size + b.size)
mean_ = (a.sum() + b.sum())/N
std_ = np.sqrt((((a - mean_)**2).sum() + ((b - mean_)**2).sum())/N)

Sample run for verification

In [266]: a = np.random.rand(3,4,2)
     ...: b = np.random.rand(2,5,3)
     ...: 

In [267]: N = float(a.size + b.size)
     ...: mean_ = (a.sum() + b.sum())/N
     ...: std_ = np.sqrt((((a - mean_)**2).sum() + ((b - mean_)**2).sum())/N)
     ...: 

In [268]: mean_
Out[268]: 0.47854757879348042

In [270]: std_
Out[270]: 0.27890341338373376

Now, to verify, let's stack and then use relevant ufuncs -

In [271]: A = np.hstack((a.ravel(), b.ravel()))

In [273]: A.mean()
Out[273]: 0.47854757879348037

In [274]: A.std()
Out[274]: 0.27890341338373376

---

List of arrays as input

For a list holding all those arrays, we need to iterate through them, like so -

A = [a,b,c] # input list of arrays

N = float(sum([i.size for i in A]))
mean_ = sum([i.sum() for i in A])/N
std_ = np.sqrt(sum([((i-mean_)**2).sum() for i in A])/N)

Sample run -

In [301]: a = np.random.rand(3,4,2)
     ...: b = np.random.rand(2,5,3)
     ...: c = np.random.rand(7,4)
     ...: 

In [302]: A = [a,b,c] # input list of arrays
     ...: N = float(sum([i.size for i in A]))
     ...: mean_ = sum([i.sum() for i in A])/N
     ...: std_ = np.sqrt(sum([((i-mean_)**2).sum() for i in A])/N)
     ...: print mean_, std_
     ...: 
0.47703535428 0.293308550786

In [303]: A = np.hstack((a.ravel(), b.ravel(), c.ravel()))
     ...: print A.mean(), A.std()
     ...: 
0.47703535428 0.293308550786