来自 scipy 的 jarque_bera 计算

Question

i am trying to calculate the Jarque-Bera-Bera test (normality test) on my data that look like that (after chain operation) :

ranking Q1  Q2  Q3  Q4
Date                
2009-12-29  nan nan nan nan
2009-12-30  0.12    -0.21   -0.36   -0.39
2009-12-31  0.05    0.09    0.06    -0.02
2010-01-01  nan nan nan nan
2010-01-04  1.45    1.90    1.81    1.77
... ... ... ... ...
2020-10-13  -0.67   -0.59   -0.63   -0.61
2020-10-14  -0.05   -0.12   -0.05   -0.13
2020-10-15  -1.91   -1.62   -1.78   -1.91
2020-10-16  1.21    1.13    1.09    1.37
2020-10-19  -0.03   0.01    0.06    -0.02

I use a function like that :

from scipy import stats

def stat(x):
    return pd.Series([x.mean(),
                      np.sqrt(x.var()),
                      stats.jarque_bera(x),
                      ],
                     index=['Return',
                            'Volatility',
                            'JB P-Value'
                            ])

data.apply(stat)

whereas the mean and variance calculation work fine, I have a error message stats.jarque_bera function with is :

ValueError: Length of passed values is 10, index implies 9.

Any idea ?

Answer 1

I tried to reproduce and the function works fine for me, by copying the 10 rows of data you are providing above. This looks like a data input issue, where some column seems to have fewer values than the index of that pd.Series (effectively somehow len(data[col]) > len(data[col].index) ). You can try to figure out which column it is by running a naive "debugging" function such as:

for col in data.columns: 
    if len(data[col].values) != len(data[col].index):
        print(f"Column {col} has more/less values than the index")

However, the Jarque-Bera test documentation on Scipy says that x can be any "array-like" structure, so you don't need to pass a pd.Series , which might run you into issues with missing values, etc. Essentially you can just pass a list of values and calculate their JB test statistic and p-value.

So with that, I would modify your function to

def stat(x):
    return pd.Series([x.mean(),
                      np.sqrt(x.var()),
                      stats.jarque_bera(x.dropna().values), # drop NaN and get numpy array instead of pd.Series
                      ],
                     index=['Return',
                            'Volatility',
                            'JB P-Value'
                            ])