![](/img/trans.png)
[英]Removing duplicates in python pandas dataframe based in column value condition
[英]Removing duplicates from Pandas dataFrame with condition for retaining original
假設我有以下數據框:
A | B
1 | Ms
1 | PhD
2 | Ms
2 | Bs
我想刪除關於A列的重復行,並且我想保留B列中值為“PhD”的行作為原始行,如果我沒有找到“PhD”,我想保留該行B列中的“Bs”。
我正在嘗試使用
df.drop_duplicates('A')
有條件
考慮使用Categoricals
。 他們很好地按非字母順序對文本進行分組/排序(除其他外。)
import pandas as pd
#create a pandas dataframe for testing with two columns A integer and B string
df = pd.DataFrame([(1, 'Ms'), (1, 'PhD'),
(2, 'Ms'), (2, 'Bs'),
(3, 'PhD'), (3, 'Bs'),
(4, 'Ms'), (4, 'PhD'), (4, 'Bs')],
columns=['A', 'B'])
print("Original data")
print(df)
# force the column's string column B to type 'category'
df['B'] = df['B'].astype('category')
# define the valid categories:
df['B'] = df['B'].cat.set_categories(['PhD', 'Bs', 'Ms'], ordered=True)
#pandas dataframe sort_values to inflicts order on your categories
df.sort_values(['A', 'B'], inplace=True, ascending=True)
print("Now sorted by custom categories (PhD > Bs > Ms)")
print(df)
# dropping duplicates keeps first
df_unique = df.drop_duplicates('A')
print("Keep the highest value category given duplicate integer group")
print(df_unique)
印刷:
Original data
A B
0 1 Ms
1 1 PhD
2 2 Ms
3 2 Bs
4 3 PhD
5 3 Bs
6 4 Ms
7 4 PhD
8 4 Bs
Now sorted by custom categories (PhD > Bs > Ms)
A B
1 1 PhD
0 1 Ms
3 2 Bs
2 2 Ms
4 3 PhD
5 3 Bs
7 4 PhD
8 4 Bs
6 4 Ms
Keep the highest value category given duplicate integer group
A B
1 1 PhD
3 2 Bs
4 3 PhD
7 4 PhD
>>> df
A B
0 1 Ms
1 1 Ms
2 1 Ms
3 1 Ms
4 1 PhD
5 2 Ms
6 2 Ms
7 2 Bs
8 2 PhD
使用自定義函數對數據框進行排序:
def sort_df(df, column_idx, key):
'''Takes a dataframe, a column index and a custom function for sorting,
returns a dataframe sorted by that column using that function'''
col = df.ix[:,column_idx]
df = df.ix[[i[1] for i in sorted(zip(col,range(len(col))), key=key)]]
return df
我們的排序功能:
cmp = lambda x:2 if 'PhD' in x else 1 if 'Bs' in x else 0
在行動:
sort_df(df,'B',cmp).drop_duplicates('A', take_last=True)
PS 在現代熊貓版本中沒有選項take_last
,請改用keep
- 請參閱文檔。
A B
4 1 PhD
8 2 PhD
假設給定 A 值的 B 值的唯一性,並且每個 A 值在 B 列中都有 B 行:
df2 = df[df['B']=="PhD"]
會給你一個包含你想要的博士行的數據框。
然后從 df 中刪除所有的博士和女士:
df = df[df['B']=="Bs"]
然后連接df和df2:
df3 = concat([df2, df])
然后你可以使用你想要的 drop_duplicates :
df3.drop_duplicates('A', inplace=True)
對您的列進行排序以將您想要保留的列放在頂部,然后drop_duplicates
做正確的事情。
import pandas as pd
df = pd.DataFrame([(1, '2022-01-25'),
(1, '2022-05-25'),
(2, '2021-12-20'),
(2, '2021-11-20'),
(3, '2020-03-03'),
(3, '2020-03-04'),
(4, '2019-07-06'),
(4, '2019-07-07'),
(4, '2019-07-05')], columns=['A', 'B'])
print("Original data")
print(df.to_string(index=False) )
#Sort your dataframe so that the one you want is on the top:
df.sort_values(['A', 'B'], inplace=True, ascending=True)
print("custom sort")
print(df.to_string(index=False) )
# dropping duplicates this way keeps first
df_unique = df.drop_duplicates('A')
print("Keep first")
print(df_unique.to_string(index=False) )
印刷:
Original data
A B
1 2022-01-25
1 2022-05-25
2 2021-12-20
2 2021-11-20
3 2020-03-03
3 2020-03-04
4 2019-07-06
4 2019-07-07
4 2019-07-05
custom sort
A B
1 2022-01-25
1 2022-05-25
2 2021-11-20
2 2021-12-20
3 2020-03-03
3 2020-03-04
4 2019-07-05
4 2019-07-06
4 2019-07-07
Keep first
A B
1 2022-01-25
2 2021-11-20
3 2020-03-03
4 2019-07-05
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.