[英]Python Pandas Dataframe idxmax is so slow. Alternatives?
I'm trying to select rows out of groups by max value using df.loc[df.groupby(keys)['column'].idxmax()]
.我正在尝试使用
df.loc[df.groupby(keys)['column'].idxmax()]
按最大值将 select 行出组。
I'm finding, however, that df.groupby(keys)['column'].idxmax()
takes a really long time on my dataset of about 27M rows.但是,我发现
df.groupby(keys)['column'].idxmax()
在我大约 27M 行的数据集上需要很长时间。 Interestingly, running df.groupby(keys)['column'].max()
on my dataset takes only 13 seconds while running df.groupby(keys)['column'].idxmax()
takes 55 minutes.有趣的是,在我的数据集上运行
df.groupby(keys)['column'].max()
只需要 13 秒,而运行df.groupby(keys)['column'].idxmax()
需要 55 分钟。 I don't understand why returning the indexes of the rows takes 250 times longer than returning a value from the row.我不明白为什么返回行的索引比从行返回值要长 250 倍。 Maybe there is something I can do to speed up idxmax?
也许我可以做些什么来加快 idxmax?
If not, is there an alternative way of selecting rows out of groups by max value that might be faster than using idxmax?如果没有,是否有另一种方法可以通过最大值从组中选择行,这可能比使用 idxmax 更快?
For additional info, I'm using two keys and sorted the dataframe on those keys prior to the groupby and idxmax operations.有关其他信息,我使用两个键并在 groupby 和 idxmax 操作之前对这些键上的 dataframe 进行排序。 Here's what it looks like in Jupyter Notebook:
这是它在 Jupyter Notebook 中的样子:
import pandas as pd
df = pd.read_csv('/data/Broadband Data/fbd_us_without_satellite_jun2019_v1.csv', encoding='ANSI', \
usecols=['BlockCode', 'HocoNum', 'HocoFinal', 'TechCode', 'Consumer', 'MaxAdDown', 'MaxAdUp'])
%%time
df = df[df.Consumer == 1]
df.sort_values(['BlockCode', 'HocoNum'], inplace=True)
print(df)
HocoNum HocoFinal BlockCode TechCode
4631064 130077 AT&T Inc. 10010201001000 10
4679561 130077 AT&T Inc. 10010201001000 11
28163032 130235 Charter Communications 10010201001000 43
11134756 131480 WideOpenWest Finance, LLC 10010201001000 42
11174634 131480 WideOpenWest Finance, LLC 10010201001000 50
... ... ... ... ...
15389917 190062 Broadband VI, LLC 780309900000014 70
10930322 130081 ATN International, Inc. 780309900000015 70
15389918 190062 Broadband VI, LLC 780309900000015 70
10930323 130081 ATN International, Inc. 780309900000016 70
15389919 190062 Broadband VI, LLC 780309900000016 70
Consumer MaxAdDown MaxAdUp
4631064 1 6.0 0.512
4679561 1 18.0 0.768
28163032 1 940.0 35.000
11134756 1 1000.0 50.000
11174634 1 1000.0 50.000
... ... ... ...
15389917 1 25.0 5.000
10930322 1 25.0 5.000
15389918 1 25.0 5.000
10930323 1 25.0 5.000
15389919 1 25.0 5.000
[26991941 rows x 7 columns]
Wall time: 21.6 s
%time df.groupby(['BlockCode', 'HocoNum'])['MaxAdDown'].max()
Wall time: 13 s
BlockCode HocoNum
10010201001000 130077 18.0
130235 940.0
131480 1000.0
10010201001001 130235 940.0
10010201001002 130077 6.0
...
780309900000014 190062 25.0
780309900000015 130081 25.0
190062 25.0
780309900000016 130081 25.0
190062 25.0
Name: MaxAdDown, Length: 20613795, dtype: float64
%time df.groupby(['BlockCode', 'HocoNum'])['MaxAdDown'].idxmax()
Wall time: 55min 24s
BlockCode HocoNum
10010201001000 130077 4679561
130235 28163032
131480 11134756
10010201001001 130235 28163033
10010201001002 130077 4637222
...
780309900000014 190062 15389917
780309900000015 130081 10930322
190062 15389918
780309900000016 130081 10930323
190062 15389919
Name: MaxAdDown, Length: 20613795, dtype: int64
You'll see in the very first rows of data there are two entries for AT&T in the same BlockCode, one for MaxAdDown of 6Mbps and one for 18Mbps.您将在第一行数据中看到,在同一个 BlockCode 中有两个 AT&T 条目,一个用于 6Mbps 的 MaxAdDown,一个用于 18Mbps。 I want to keep the 18Mbps row and drop the 6Mbps row, so that there is one row per company per BlockCode that has the the maximum MaxAdDown value.
我想保留 18Mbps 行并删除 6Mbps 行,以便每个 BlockCode 的每个公司都有一行具有最大 MaxAdDown 值。 I need the entire row, not just the MaxAdDown value.
我需要整行,而不仅仅是 MaxAdDown 值。
sort and drop duplicates:排序并删除重复项:
df.sort('MaxAdDown').drop_duplicates(['BlockCode', 'HocoNum'], keep='last')
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.