根据日期条件从 dataframe 中删除记录

Question

I have dataframe with a column named date, it contains the following dates:

In [67]: df.date.drop_duplicates()
Out[67]: 
0      2020-02-04
570    2020-02-19
1157   2020-03-03
1791   2020-04-02
2452   2020-04-08
3113   2020-05-05
3777   2020-06-03
4445   2020-07-02
5131   2020-08-04
Name: date, dtype: datetime64[ns]

I only want to have monthly data. And from the monthly data I want to keep the earliest of this month. So here I want to delete all records where the date is 2020-02-19 and 2020-04-08. The problem is that I never know which dates I will recieve. I could have also recieved 2020-07-22 as well - then I would have wanted to delete all records with date 2020-07-22 too as I alreaedy have 2020-07-02.

Do you know a smooth way to do that? I thought of sorting the values in a way that they look like this:

2020-02-04
2020-03-03
2020-04-02
2020-05-05
2020-06-03
2020-07-02
2020-08-04
2020-02-19
2020-04-08

Then I could delete all records where the date is one of the dates after the 7th row (counted from 1), as I always have a variable that determines how many datapoints I need. But I couldn't figure out how to sort it like that. Do you know any other way or could help me sorting the date values? Thank you so much!

Answer 1

IIUC, you can do a groupby month, and then get the min :

df.groupby(df.date.dt.month).min()

If 'date' has more than one year, group by year and month:

df.groupby([df.date.dt.month,df.date.dt.year]).min()

Output:

           date
           
2    2020-02-04
3    2020-03-03
4    2020-04-02
5    2020-05-05
6    2020-06-03
7    2020-07-02
8    2020-08-04

Answer 2

This works even if your data is longer than a year:

df.sort_values(by='date').groupby(df.date.dt.year.astype(str)
                                  + df.date.dt.month.astype(str)).first()

Output:

0       idx       date
date                  
20202     0 2020-02-04
20203  1157 2020-03-03
20204  1791 2020-04-02
20205  3113 2020-05-05
20206  3777 2020-06-03
20207  4445 2020-07-02
20208  5131 2020-08-04