How to append into from one dataframe into another empty one based on two criteria's?

Question

I am storing an API call into a dataframe(successfully) called df1. The API call will be ran twice a day. I need to append df1 into df2 where the following conditions are met:

1. append rows from df1 into df2 if ID in df1 is not present in df2
2. append rows from df1 into df2 if column 'Updated_Date' in df1 is greater than todays date in df2. 

The script will pull data from the API twice a day, so on the first day of running the script there is no data in df2 but everyday after the first day there will be records in df1 from the API. So on day1 df1 and df2 should be equal.

Here is how I am getting today's date:

import datetime as dt
benchmark_date = dt.datetime.today().strftime("%m/%d/%y")

Here is how I am appending into an empty dataframe

df2 = df2.append(df1)

I am not sure from here how to build the logic in the two steps above.

df1 structure:

ID           Updated_Date
0            2/14/15
1            2/20/15

Thank you in advance.

Answer 1

Assuming there is data in d1 and d2 this should satisfy your conditions. This will create a boolean filter to only select rows that satisfy your conditions. I would probably change the format of your datetime string to be YYYY-MM-DD so that string comparison works correctly. You could also just keep it in datetime format. If you need to convert string column in your dataframe you can do so by using pd.to_datetime .

benchmark_date = dt.datetime.today().strftime("%Y-%m-%d")

# this filter will satisfy the conditions 1 and 2
df1_filter = ((~df1.ID.isin(df2.ID.unique())) & (df1.Updated_Date > benchmark_date))
# concatenate your original dataframe with new filtered dataframe
df = pd.concat([df2, df1[df1_filter]])

# you can save df which is your combined dataframe