Python Pandas：合并返回 Nan

Question

I have two dataframes ie df1 and df2. df1 is

df1

       date     time
0   2015-04-01  00:00:00
1   2015-04-01  00:30:00
2   2015-04-01  01:00:00
3   2015-04-01  01:30:00
4   2015-04-01  02:00:00

Datatype of df1 is:

date             object
time    timedelta64[ns]
dtype: object

df2 is:

     INCIDENT_TIME  INTERRUPTION_TIME      MINUTES
0   2015-01-08         03:00:00             1056.0
1   2015-01-10         23:30:00             3234.0
2   2015-04-01         01:00:00             3712.0
3   2015-04-01         01:30:00             3045.0
4   2015-04-01         02:00:00             525.0

datatype of df2 is 
INCIDENT_TIME                 object
INTERRUPTION_TIME    timedelta64[ns]
MINUTES               float64
dtype: object

I want to do left merge. So my code is:

final_df= pd.merge(df1,df2,left_on=['date','time'],right_on=['INCIDENT_TIME','INTERRUPTION_TIME'],how='left')

However, it does not produce the desired output. The output was:

       date      time     INCIDENT_TIME   INTERRUPTION_TIME  CONSUM_MINUTES
0   2015-04-01  00:00:00    NaN               NaT                NaN
1   2015-04-01  00:30:00    NaN               NaT                NaN
2   2015-04-01  01:00:00    NaN               NaT                NaN
3   2015-04-01  01:30:00    NaN               NaT                NaN
4   2015-04-01  02:00:00    NaN               NaT                NaN 

For diagnostic purpose, I used inner join and the output was blank/null. Intially I thought the difference of datatypes might be causing the issue. So I changed the datatype of time of df1 and INTERRUPTION_TIME of df2 to str. Now datatype of both dataframes are :

df1
date    object
time    object
dtype: object

df2
INCIDENT_TIME         object
INTERRUPTION_TIME     object
MINUTES               float64
dtype: object

When I ran the program again, it returned the same output. I am not sure where am I making the mistake. Could anyone help me in fixing the issue please?

Answer 1

i think you need to convert to datetime:

import pandas as _pd

df1['date'] = _pd.to_datetime(df1['date'])


print(df1.dtypes)

df2['INCIDENT_TIME'] = _pd.to_datetime(df2['INCIDENT_TIME'])
print(df2.dtypes)

final_df= _pd.merge(df1,df2,left_on=['date','time'],right_on=['INCIDENT_TIME','INTERRUPTION_TIME'],how='left')
print(final_df)

Which gives result:

        date      time INCIDENT_TIME INTERRUPTION_TIME  MINUTES
0 2015-04-01  00:00:00           NaT               NaN      NaN
1 2015-04-01  00:30:00           NaT               NaN      NaN
2 2015-04-01  01:00:00    2015-04-01          01:00:00   3712.0
3 2015-04-01  01:30:00    2015-04-01          01:30:00   3045.0
4 2015-04-01  02:00:00    2015-04-01          02:00:00    525.0

I would ideeally create a full datetime column in order to make sure that the match corresponds for sure with date and time, which would look like this:

import pandas as _pd


df1['datetime'] = _pd.to_datetime(df1['date']+ ' ' + df1['time'], format='%Y-%m-%d %H:%M:%S')

print(df1)


df2['incident_datetime'] = _pd.to_datetime(df2['INCIDENT_TIME']+ ' ' + df2['INTERRUPTION_TIME'], format='%Y-%m-%d %H:%M:%S')
final_df = _pd.merge(df1,df2,left_on=['datetime'],right_on=['incident_datetime'],how='left')

#dropping none matching columns
final_df = final_df.dropna()

print(final_df)

Which gives the following results:

         date      time  ... MINUTES   incident_datetime
2  2015-04-01  01:00:00  ...  3712.0 2015-04-01 01:00:00
3  2015-04-01  01:30:00  ...  3045.0 2015-04-01 01:30:00
4  2015-04-01  02:00:00  ...   525.0 2015-04-01 02:00:00

great reference for datetime convertion (which is strptime not strftime) with panda: https://www.journaldev.com/23365/python-string-to-datetime-strptime

Answer 2

The data may contain white-space or other characters, you can try calling the strip function on every cell this might solve the issues.

Use datetime64[ns] for date columns for better results.