矩阵归一化

Question

I have a 150x4 matrix X which I created from a pandas dataframe using the following code:

X = df_new.as_matrix()

I have to normalize it using this function:

I know that Uj is the mean val of j , and that σ j is the standard deviation of j , but I don't understand what j is. I'm having a little trouble understanding what the bar on X is, and I'm confused by the commas in the equation (I don't know if they have any significance or not).

Can anyone help me understand what this equation means so I can then write the normalization using sklearn?

Answer 1

The indexes for matrix X are row (i) and column (j). Hence, X,j means column j of matrix X . Ie normalize each column of matrix X to z-scores .

You can do that using pandas:

df_new_zscores = (df_new - df_new.mean()) / df_new.std()

Answer 2

You don't actually need to write code for the normalization yourself - it comes ready with sklearn.preprocessing.scale .

Here is an example from the docs :

>>> from sklearn import preprocessing
>>> import numpy as np
>>> X_train = np.array([[ 1., -1.,  2.],
...                     [ 2.,  0.,  0.],
...                     [ 0.,  1., -1.]])
>>> X_scaled = preprocessing.scale(X_train)

>>> X_scaled                                          
array([[ 0.  ..., -1.22...,  1.33...],
       [ 1.22...,  0.  ..., -0.26...],
       [-1.22...,  1.22..., -1.06...]])

When used with the default setting axis=0 , the mormalization happens column-wise (ie for each column j , as in your equestion). As a result, it is easy to confirm that scaled data has zero mean and unit variance:

>>> X_scaled.mean(axis=0)
array([ 0.,  0.,  0.])

>>> X_scaled.std(axis=0)
array([ 1.,  1.,  1.])

Answer 3

I do not know pandas but I think that the equation means that the normalized matrix is given by You subtract the empirical mean and devide by the empirical standard deviation per column.

You sometimes use this for Principal Component Analysis.