[英]Fastest way to filter a pandas dataframe on multiple columns
I have a pandas dataframe with several columns that labels data in a final column, for example, 我有一个带有多个列的pandas数据框,它们在最后一列中标记了数据,例如,
df = pd.DataFrame( {'1_label' : ['a1','b1','c1','d1'],
'2_label' : ['a2','b2','c2','d2'],
'3_label' : ['a3','b3','c3','d3'],
'data' : [1,2,3,4]})
df = 1_label 2_label 3_label data
0 a1 a2 a3 1
1 b1 b2 b3 2
2 c1 c2 c3 3
3 d1 d2 d3 4
and a list of tuples, 以及元组列表
list_t = [('a1','a2','a3'), ('d1','d2','d3')]
I want to filter this dataframe and return a new dataframe containing only the rows that correspond to the tuples in my list. 我想过滤此数据框并返回一个仅包含与列表中元组相对应的行的新数据框。
result = 1_label 2_label 3_label data
0 a1 a2 a3 1
1 d1 d2 d3 4
My naive (and C++ inspired) solution was to use append (like vector::push_back) 我的幼稚(和C ++启发)解决方案是使用append(例如vector :: push_back)
for l1, l2, l3 in list_t:
if df[(df['1_label'] == l1) &
(df['2_label'] == l2) &
(df['3_label'] == l3)].empty is False:
result = result.append(df[(df['1_label'] == l1) &
(df['2_label'] == l2) &
(df['3_label'] == l3)]
While my solution works I suspect it is horrendously slow for large dataframes and large list of tuples as I think pandas creates a new dataframe upon each call to append. 虽然我的解决方案有效,但我怀疑大数据框和大元组列表的速度太慢了,因为我认为熊猫会在每次追加请求时创建一个新的数据框。 Could anyone suggest a faster/cleaner way to do this?
有人可以建议一种更快/更清洁的方法吗? Thanks!
谢谢!
If I understood correctly, merge should do the job: 如果我正确理解,则merge应该可以完成以下工作:
pd.DataFrame(list_t, columns=['1_label', '2_label', '3_label']).merge(df)
Out[73]:
1_label 2_label 3_label data
0 a1 a2 a3 1
1 d1 d2 d3 4
Assuming no duplicates, you could create index out of the columns you want to "filter" on: 假设没有重复项,则可以在要“过滤”的列上创建索引:
In [10]: df
Out[10]:
1_label 2_label 3_label data
0 a1 a2 a3 1
1 b1 b2 b3 2
2 c1 c2 c3 3
3 d1 d2 d3 4
In [11]: df.set_index(['1_label', '2_label', '3_label'])\
.loc[[('a1','a2','a3'), ('d1','d2','d3')]]\
.reset_index()
Out[11]:
1_label 2_label 3_label data
0 a1 a2 a3 1
1 d1 d2 d3 4
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.