从 Pandas 数据框中删除重复项，同时保留多数元素

Question

I have a pandas Dataframe that looks like this:

   Cat  Date
1  A    2019-12-30
2  A    2019-12-30
3  A    2020-12-30
4  A    2020-01-06
5  A    2020-01-06
6  B    2020-01-06
7  B    2020-01-13
8  B    2020-01-13
9  A    2020-01-13
 .    .
 .    .
 .    .

There are duplicate dates in the Date column, and I want to "smush" down the DataFrame so that all the duplicate dates are removed. However, to determine what's in the "Cat" column after this "smushing", I want to pick the majority element of the dates that are being "smushed".

Thus, I want the output to be:

   Cat  Date
1  A    2019-12-30
2  A    2020-01-06
3  B    2020-01-13
 .    .
 .    .
 .    .

Efficiency is important, I want to be able to do this as quickly as possible, as my DataFrame is quite large (100k rows). There is a guarantee that the number of repeated dates will always be odd, and that the total number of different "Cat" letters can be max 2, so there is no concern of ties.

Answer 1

Try value_counts to count all values after a groupby on the date column:

df.groupby("Date").agg(lambda x: x.value_counts().index[0])
#            Cat
# Date
# 2019-12-30   A
# 2020-01-06   A
# 2020-01-13   B
# 2020-12-30   A

Explanations :

1. Split the dataframe in groups according the Date using groupby .

2. Apply an aggregation using agg . This function accepts an function to aggregate the groups.

3. Define the aggregation function:

   3.1. Get the number of values per group using the value_counts function:

print(df.groupby("Date").agg(lambda x: x.value_counts()))
#                Cat
# Date
# 2019-12-30       2
# 2020-01-06  [3, 2]
# 2020-01-13  [2, 1]
# 2020-12-30       1

Note: the result from the value_counts method is an ordered series.

3.2. However, we actually want the values and not the count . The trick is to use the index on the serie.

print(df.groupby("Date").agg(lambda x: x.value_counts().index))
#                Cat
# Date
# 2019-12-30       A
# 2020-01-06  [B, A]
# 2020-01-13  [B, A]
# 2020-12-30       A

3.3. Finaly, select the first value:

print(df.groupby("Date").agg(lambda x: x.value_counts().index[0]))
#            Cat
# Date
# 2019-12-30   A
# 2020-01-06   B
# 2020-01-13   B
# 2020-12-30   A

Answer 2

Here's a simple solution

def removeDuplicatesKeepBest(df):
    # sort the data frame 
    df.sort_values(by="Cat")
    # Look only in the date column and only keep the first occurence if there is a dulplicate
    df.drop_duplicates(subset = "Date" , keep = 'first', inplace = True)

    return df

Hope this helps!

Answer 3

I'd consider the old groupby

df.groupby(["Cat", "Date"]).size()\
  .reset_index(name="to_drop")\
  .drop("to_drop", axis=1)

Alternatively you can use drop duplicates with two columns

df.drop_duplicates(['Date',"Cat"])