Python Pandas: How do I compare the date in the last row of a Dataframe to a series of dates in another data frame

Question

I am trying to bring in 2 excel files. I have converted them to dataframes and changed the dateCreated columns in both files to datatype datetime.

Now I want to take file 2's last row, find the date in the 'dateCreated' column, then compare that to the 'dateCreated' dates in File 1. Then I want to take all of the rows of data(from File 1) that have a date after the last rows date from file 2 and paste them into file 2.

I have tried several ways to do this, and I keep running across errors about the datatype or that I can't compare a series in a dataframe to a non series.

This is the code I have currently after several iterations and its not even close to correct:

# Create Pandas Dataframes
df = pd.read_excel(BASE_DIR+r'\www\First_File.xlsx')
df2 = pd.read_excel(BASE_DIR+r'\www\Second_File.xlsx')

# Convert Dates on First File to DateTime Format
df['dateCreated'] = pd.to_datetime(df['dateCreated'], format='%m/%d/%Y')

# Calculate the date of last row in Second File
lastRowOfDF2 = df2.iloc[-1:]
lastRowRange = pd.to_datetime(lastRowOfDF2['dateCreated'], format='m%/%d/%Y')

for x in df['dateCreated']:

    if lastRowRange['dateCreated'] < x:

        print(x)

This code gives me a datatype error. I'm not sure what I'm supposed to convert the columns to, in order to compare the dates at this point.

Answer 1

I suppose if the dateCreated Column in both the dataframes has the same datatype then you don't need to change the datatype to datetime. You can simply use pandas dataframe filtering to get the desired result in the following way.

# Read the data
df = pd.read_excel(BASE_DIR+r'\www\First_File.xlsx')
df2 = pd.read_excel(BASE_DIR+r'\www\Second_File.xlsx')

# Get the date in the last row of df2
last_row_date = df2.loc[-1, 'dateCreated']

desired_data = df[df.dateCreated > last_row_date]

I hope this solves your problem.