Dataframe : Group by day and set values in a specific column after the first occurrence

Question

I have this dataframe:

                        column1     column2 column3     Filter
2000/01/02 13:35        2.55651     0.1275  0.198       1.0
2000/01/03 14:35        3.30585     0.9425  0.009       1.0
2000/01/04 16:30        3.40865     1.7897  0.515       1.0
2000/01/05 14:15        2.96273     0.6266  0.506       1.0
2000/01/07 14:40        2.75470     0.1724  0.405       1.0
2000/01/07 15:40        2.50288     0.4133  0.075       **1.0**
2000/01/09 14:35        2.20984     0.7232  0.818       1.0
2000/01/09 16:00        2.21001     0.2815  0.160       **1.0**

and I want to set to zero the duplicates of the "filter" column after the first occurence inside each day. In this example I put ** in the values I want to set them to zero. The others are ok. I'd like use loc function. Thanks in advance.

Answer 1

You can get this by making a temporary column that just has the date (not the whole datetime) and then use df.duplicated() to mark the subsequent occurrences of the date in another temporary column, and then use that flag to drive changing the Filter column

import pandas as pd
df = pd.read_csv('testdf.csv') #put your real filepath here
df.rename(columns={'Unnamed: 0':'datetime'},inplace=True) # I renamed that first column, just for fun
df['datetime'] = pd.to_datetime(df['datetime'],format='%Y/%m/%d %H:%M') #changes the format from a string to a datetime
df['date'] = pd.to_datetime(df['datetime']).dt.date # makes a new column with jsut the date
df['Duplicate?'] = df.duplicated(subset='date',keep='first') #makes a new column called duplicated. first instance is False and all next instances are True
df.loc[df['Duplicate?'] == True,'Filter'] = 0 #changes Filter to 0 when Duplicated is True
df = df.drop(columns=['date','Duplicate?']) #drops our temporary date and Duplicated columns
print(df)