Consider the following solution to computing a within-group diff in Pandas:
df = df.set_index(['ticker', 'date']).sort_index()[['value']]
df['diff'] = np.nan
idx = pd.IndexSlice
for ix in df.index.levels[0]:
df.loc[ idx[ix,:], 'diff'] = df.loc[idx[ix,:], 'value' ].diff()
For:
> df
date ticker value
0 63 C 1.65
1 88 C -1.93
2 22 C -1.29
3 76 A -0.79
4 72 B -1.24
5 34 A -0.23
6 92 B 2.43
7 22 A 0.55
8 32 A -2.50
9 59 B -1.01
It returns:
> df
value diff
ticker date
A 22 0.55 NaN
32 -2.50 -3.05
34 -0.23 2.27
76 -0.79 -0.56
B 59 -1.01 NaN
72 -1.24 -0.23
92 2.43 3.67
C 22 -1.29 NaN
63 1.65 2.94
88 -1.93 -3.58
The solution does not scale well for large dataframes. It takes minutes for a dataframe with a shape (405344,2)
. This is presumably the case because I am iterating through each value for the first level in the main loop.
Is there any way of speeding this up in Pandas? Is looping through index values a good way of solving this problem? Could numba
perhaps be used for this?
Here's another way, which ought to be a lot faster.
First, sort based on ticker and date:
In [11]: df = df.set_index(['ticker', 'date']).sort_index()
In [12]: df
Out[12]:
value
ticker date
A 22 0.55
32 -2.50
34 -0.23
76 -0.79
B 59 -1.01
72 -1.24
92 2.43
C 22 -1.29
63 1.65
88 -1.93
Add the diff column:
In [13]: df['diff'] = df['value'].diff()
To fill in the NaNs, we can find the first line as follows (there may be a nicer way):
In [14]: s = pd.Series(df.index.labels[0])
In [15]: s != s.shift()
Out[15]:
0 True
1 False
2 False
3 False
4 True
5 False
6 False
7 True
8 False
9 False
dtype: bool
In [16]: df.loc[(s != s.shift()).values 'diff'] = np.nan
In [17]: df
Out[17]:
value diff
ticker date
A 22 0.55 NaN
32 -2.50 -3.05
34 -0.23 2.27
76 -0.79 -0.56
B 59 -1.01 NaN
72 -1.24 -0.23
92 2.43 3.67
C 22 -1.29 NaN
63 1.65 2.94
88 -1.93 -3.58
Using groupby/apply is simple and elegant, but it can be slow in Pandas. Bodo JIT compiler (based on Numba) can make it fast in many cases:
pip install bodo
import pandas as pd
import numpy as np
import bodo
def value_and_diff(subdf):
subdf = subdf.set_index('date').sort_index()
return pd.DataFrame({'value': subdf['value'],
'diff': subdf['value'].diff()})
@bodo.jit(distributed=False)
def f(df):
df2 = df.groupby('ticker').apply(value_and_diff)
return df2
np.random.seed(0)
df = pd.DataFrame({'ticker': ["A", "B", "C", "D"] * 25_000,
'date': pd.date_range('1/1/2000', periods=100_000, freq='T'),
'value': np.random.randn(100_000)})
print(f(df))
As an alternative, you could do the sorting and indexing within each group. Though not time-tested yet:
In [11]: def value_and_diff(subdf):
subdf = subdf.set_index('date').sort_index()
return pd.DataFrame({'value': subdf['value'],
'diff': subdf['value'].diff()})
In [12]: df.groupby('ticker').apply(value_and_diff)
Out[12]:
diff value
ticker date
A 22 NaN 0.55
32 -3.05 -2.50
34 2.27 -0.23
76 -0.56 -0.79
B 59 NaN -1.01
72 -0.23 -1.24
92 3.67 2.43
C 22 NaN -1.29
63 2.94 1.65
88 -3.58 -1.93
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.