[英]Speeding up group-wise differencing in Pandas
Consider the following solution to computing a within-group diff in Pandas:考虑以下计算 Pandas 中组内差异的解决方案:
df = df.set_index(['ticker', 'date']).sort_index()[['value']]
df['diff'] = np.nan
idx = pd.IndexSlice
for ix in df.index.levels[0]:
df.loc[ idx[ix,:], 'diff'] = df.loc[idx[ix,:], 'value' ].diff()
For:为了:
> df
date ticker value
0 63 C 1.65
1 88 C -1.93
2 22 C -1.29
3 76 A -0.79
4 72 B -1.24
5 34 A -0.23
6 92 B 2.43
7 22 A 0.55
8 32 A -2.50
9 59 B -1.01
It returns:它返回:
> df
value diff
ticker date
A 22 0.55 NaN
32 -2.50 -3.05
34 -0.23 2.27
76 -0.79 -0.56
B 59 -1.01 NaN
72 -1.24 -0.23
92 2.43 3.67
C 22 -1.29 NaN
63 1.65 2.94
88 -1.93 -3.58
The solution does not scale well for large dataframes.该解决方案不适用于大型数据帧。 It takes minutes for a dataframe with a shape
(405344,2)
.形状为
(405344,2)
的 dataframe 需要几分钟。 This is presumably the case because I am iterating through each value for the first level in the main loop.大概是这种情况,因为我正在遍历主循环中第一级的每个值。
Is there any way of speeding this up in Pandas?在 Pandas 中有什么方法可以加快速度吗? Is looping through index values a good way of solving this problem?
遍历索引值是解决这个问题的好方法吗? Could
numba
perhaps be used for this? numba
可以用于此吗?
Here's another way, which ought to be a lot faster. 这是另一种方法,应该更快一些。
First, sort based on ticker and date: 首先,根据代码和日期排序:
In [11]: df = df.set_index(['ticker', 'date']).sort_index()
In [12]: df
Out[12]:
value
ticker date
A 22 0.55
32 -2.50
34 -0.23
76 -0.79
B 59 -1.01
72 -1.24
92 2.43
C 22 -1.29
63 1.65
88 -1.93
Add the diff column: 添加差异列:
In [13]: df['diff'] = df['value'].diff()
To fill in the NaNs, we can find the first line as follows (there may be a nicer way): 要填写NaN,我们可以找到第一行,如下所示(可能会有更好的方法):
In [14]: s = pd.Series(df.index.labels[0])
In [15]: s != s.shift()
Out[15]:
0 True
1 False
2 False
3 False
4 True
5 False
6 False
7 True
8 False
9 False
dtype: bool
In [16]: df.loc[(s != s.shift()).values 'diff'] = np.nan
In [17]: df
Out[17]:
value diff
ticker date
A 22 0.55 NaN
32 -2.50 -3.05
34 -0.23 2.27
76 -0.79 -0.56
B 59 -1.01 NaN
72 -1.24 -0.23
92 2.43 3.67
C 22 -1.29 NaN
63 1.65 2.94
88 -1.93 -3.58
Using groupby/apply is simple and elegant, but it can be slow in Pandas. Bodo JIT compiler (based on Numba) can make it fast in many cases:使用 groupby/apply 简单而优雅,但在 Pandas 中可能会很慢。Bodo JIT 编译器(基于 Numba)在许多情况下可以使其变得很快:
pip install bodo
import pandas as pd
import numpy as np
import bodo
def value_and_diff(subdf):
subdf = subdf.set_index('date').sort_index()
return pd.DataFrame({'value': subdf['value'],
'diff': subdf['value'].diff()})
@bodo.jit(distributed=False)
def f(df):
df2 = df.groupby('ticker').apply(value_and_diff)
return df2
np.random.seed(0)
df = pd.DataFrame({'ticker': ["A", "B", "C", "D"] * 25_000,
'date': pd.date_range('1/1/2000', periods=100_000, freq='T'),
'value': np.random.randn(100_000)})
print(f(df))
As an alternative, you could do the sorting and indexing within each group. 或者,您可以在每个组中进行排序和索引。 Though not time-tested yet:
虽然尚未经过时间测试:
In [11]: def value_and_diff(subdf):
subdf = subdf.set_index('date').sort_index()
return pd.DataFrame({'value': subdf['value'],
'diff': subdf['value'].diff()})
In [12]: df.groupby('ticker').apply(value_and_diff)
Out[12]:
diff value
ticker date
A 22 NaN 0.55
32 -3.05 -2.50
34 2.27 -0.23
76 -0.56 -0.79
B 59 NaN -1.01
72 -0.23 -1.24
92 3.67 2.43
C 22 NaN -1.29
63 2.94 1.65
88 -3.58 -1.93
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.