如何将列添加到 pandas DataFrame，其值基于两个 DataFrame 中的匹配值

Question

I am working with two pandas DataFrames. One contains the performance data of different servers for every hour and looks something like this:

Date	time	server_name	CPU	Memory
2020-10-25	300	server1	90.2	64.4
2020-10-25	300	server2	50.4	23.3

In this case, '300' in the column 'time' means 3am.

The second DataFrame contains data to errors for the different servers and looks something like this:

server_name	timestamp
server1	2020-10-25 00:45:04
server2	2020-10-25 03:45:04

I would like to have a column added to the first DataFrame with the performance metrics, which indicates for every server for every hour if an error occurred at this time. Please note that an error which occurred at 3:45am should be assigned to the row for 3am for the respective server. It should look something like this:

Date	time	server_name	CPU	Memory	error
2020-10-25	300	server1	90.2	64.4	0
2020-10-25	300	server2	50.4	23.3	1

In this case, '1' in the column 'error' would mean that at this time, an error occurred on the server.

I already tried merging the DataFrames on date, time and server_name and many other approaches, but I just don't get the desired results.

Answer 1

Assuming df1 is your first dataframe, and df2 is the second one, you could add a timestamp column to df1 by adding the Date and time column, and then use merge_asof to bind each row for the second frame to a row from that modified dataframe.

From there, you could merge that new data frame into the first one, and a groupby and count should give the expected result.

Possible code:

df3 = pd.merge_asof(df2, df1.assign(timestamp=df1['Date']
                                    + pd.to_timedelta(df1['time']/100, 'H')),
                    by='server_name', on='timestamp',
                    tolerance=pd.Timedelta('1H'))

print(df3)

result = df1.merge(df3[['server_name', 'timestamp', 'Date', 'time']], 'left',
                   on=['server_name', 'Date', 'time']
                   ).groupby(['server_name',  'Date', 'time', 'CPU', 'Memory']
                             ).count().rename(columns={'timestamp': 'error'}
                                              ).reset_index()

With your data, it gives as expected:

  server_name       Date  time   CPU  Memory  error
0     server1 2020-10-25   300  90.2    64.4      0
1     server2 2020-10-25   300  50.4    23.3      1