[英]Calculating Autocorrelation of Pandas DataFrame along each Column
I want to calculate the autocorrelation coefficients of lag length one among columns of a Pandas DataFrame. A snippet of my data is:我想计算 Pandas DataFrame 的列中滞后长度之一的自相关系数。我的数据片段是:
RF PC C D PN DN P
year
1890 NaN NaN NaN NaN NaN NaN NaN
1891 -0.028470 -0.052632 0.042254 0.081818 -0.045541 0.047619 -0.016974
1892 -0.249084 0.000000 0.027027 0.067227 0.099404 0.045455 0.122337
1893 0.653659 0.000000 0.000000 0.039370 -0.135624 0.043478 -0.142062
Along year , I want to calculate autocorrelations of lag one for each column ( RF , PC , etc...).沿着year ,我想计算每列( RF , PC等)的滞后一个的自相关。
To calculate the autocorrelations, I extracted two time series for each column whose start and end data differed by one year and then calculated correlation coefficients with numpy.corrcoef
.为了计算自相关,我为开始和结束数据相差一年的每一列提取了两个时间序列,然后使用numpy.corrcoef
计算相关系数。
For example, I wrote:例如,我写道:
numpy.corrcoef(data[['C']][1:-1],data[['C']][2:])
(the entire DataFrame is called data
). (整个 DataFrame 称为data
)。
However, the command unfortunately returned:然而,该命令不幸返回:
array([[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan],
...,
[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan],
[ nan, nan, nan, ..., nan, nan, nan]])
Can somebody kindly advise me on how to calculate autocorrelations?有人可以建议我如何计算自相关吗?
This is a late answer, but for future users, you can also use the pandas.Series.autocorr(), which calculates lag-N (default=1) autocorrelation on Series: 这是一个迟到的答案,但对于未来的用户,您还可以使用pandas.Series.autocorr(),它计算Series上的lag-N(默认值= 1)自相关:
df['C'].autocorr(lag=1)
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.autocorr.html#pandas.Series.autocorr http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.autocorr.html#pandas.Series.autocorr
.autocorr
applies to Series, not DataFrames. .autocorr
适用于Series,而不适用于DataFrames。 You can use .apply
to apply to a DataFrame: 您可以使用.apply
应用于DataFrame:
def df_autocorr(df, lag=1, axis=0):
"""Compute full-sample column-wise autocorrelation for a DataFrame."""
return df.apply(lambda col: col.autocorr(lag), axis=axis)
d1 = DataFrame(np.random.randn(100, 6))
df_autocorr(d1)
Out[32]:
0 0.141
1 -0.028
2 -0.031
3 0.114
4 -0.121
5 0.060
dtype: float64
You could also compute rolling autocorrelations with a specified window as follows (this is what .autocorr is doing under the hood): 您还可以使用指定窗口计算滚动自相关,如下所示(这是.autocorr在幕后进行的操作):
def df_rolling_autocorr(df, window, lag=1):
"""Compute rolling column-wise autocorrelation for a DataFrame."""
return (df.rolling(window=window)
.corr(df.shift(lag))) # could .dropna() here
df_rolling_autocorr(d1, window=21).dropna().head()
Out[38]:
0 1 2 3 4 5
21 -0.173 -0.367 0.142 -0.044 -0.080 0.012
22 0.015 -0.341 0.250 -0.036 0.023 -0.012
23 0.038 -0.329 0.279 -0.026 0.075 -0.121
24 -0.025 -0.361 0.319 0.117 0.031 -0.120
25 0.119 -0.320 0.181 -0.011 0.038 -0.111
you should use: 你应该使用:
numpy.corrcoef(df['C'][1:-1], df['C'][2:])
df[['C']]
represents a dataframe with only one column, while df['C']
is a series containing the values in your C column. df[['C']]
表示只有一列的数据帧,而df['C']
是包含C列中值的系列。
As I believe the use case where we need a window corresponding to highest correlation is more common, I have added another function which returns that window length per feature.因为我相信我们需要对应于最高相关性的 window 的用例更为常见,所以我添加了另一个 function,它返回每个特征的 window 长度。
# Find autocorrelation example.
def df_autocorr(df, lag=1, axis=0):
"""Compute full-sample column-wise autocorrelation for a DataFrame."""
return df.apply(lambda col: col.autocorr(lag), axis=axis)
def df_rolling_autocorr(df, window, lag=1):
"""Compute rolling column-wise autocorrelation for a DataFrame."""
return (df.rolling(window=window)
.corr(df.shift(lag))) # could .dropna() here
def df_autocorr_highest(df, window_min, window_max, lag_f):
"""Returns a dictionary containing highest correlation coefficient wrt window length."""
df_corrs = pd.DataFrame()
df_corr_dict = {}
for i in range(len(df.columns)):
corr_init = 0
corr_index = 0
for j in range(window_min, window_max):
corr = df_rolling_autocorr(df.iloc[:,i], window=j, lag=lag_f).dropna().mean()
if corr > corr_init:
corr_init = corr
corr_index = j
corr_label = df.columns[i] + "_corr"
df_corr_dict[corr_label] = [corr_init, corr_index]
return df_corr_dict
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.