遍历每一行时提高代码效率：Pandas Dataframe

Question

The code below, calculates the duration and distance between two dataframes and if the duration and distance is less than a specific amount , a value is appended to a new dataframe.

The code below is computationally expensive especially for a large dataframe.

Linked_df=pd.DataFrame()
#for each unique date
for unq_date in R_Unique_Dates:
    #print('unq_dat: ',unq_date)
    #obtain dataframe of Mi and Ri of a specific date
    #add a column for index to track orignal index
    M=Mi.loc[(pd.to_datetime(Mi ['EventDate']) == unq_date) ]
    R=Ri.loc[(pd.to_datetime(Ri['EventDate']) == unq_date) ]
    #Check if unique date exist in M
    if ( M.empty==False) :
        for indexR, rowR in R.iterrows():
            #get duration 
            for indexM, rowM in M.iterrows():

                        duration=datetime.combine(date.today(), rowR['EventTime']) - datetime.combine(date.today(), rowM['EventTime'])
                        dayys = duration.days
                        if (duration.days < 0):
                            duration=datetime.combine(date.today(), rowM['EventTime']) - datetime.combine(date.today(), rowR['EventTime'])
                            dayis = duration.days

                        hours, remainder = divmod(duration.seconds, 3600)
                        minutes, seconds = divmod(remainder, 60) 
                        if (hours==0)&(minutes==0)&(seconds<11):
                            range_15m=dist_TwoPoints_LatLong(rowR['lat_t'],rowR['lon_t'],rowM['lat'],rowM['long'])
                            #print(range_15m)
                            if (range_15m <15):
                                #append in new dataframe 
                                rowM['y']=rowR['y']
                                row1 = pd.DataFrame(rowM)
                                row1 = row1.transpose()
                                Linked_df= pd.concat([Linked_df, row1], ignore_index=True)

Suppose the data in Mi and Ri are the following:

Ri Dataset

lat_t   lon_t   y   speed_t sprung_weight_t duration_capture    EventDate   EventTime
-27.7816    22.9939 4   27.1    442.0   2.819999933242798   2017/11/01  12:09:15
-27.7814    22.9939 3   27.3    447.6   2.8359999656677246  2017/11/01  12:09:18
-27.7812    22.9939 3   25.4    412.2   2.884000062942505   2017/11/01  12:09:21
-27.7809    22.994  3   26.1    413.6   2.9670000076293945  2017/11/01  12:09:23
-27.7807    22.9941 3   25.4    395.0   2.938999891281128   2017/11/01  12:09:26
-27.7805    22.9941 3   21.7    451.9   3.2829999923706055  2017/11/01  12:09:29
-27.7803    22.9942 3   20.2    441.7   3.6730000972747803  2017/11/01  12:09:33
-27.7801    22.9942 4   16.7    443.3   4.25                2017/11/01  12:09:36
-27.7798    22.9942 3   15.4    438.2   4.819000005722046   2017/11/01  12:09:41
-27.7796    22.9942 3   15.4    436.1   5.0309998989105225  2017/11/01  12:09:45
-27.7794    22.9942 4   15.8    451.6   5.232000112533569   2017/11/01  12:09:50
-27.7793    22.9941 3   18.2    439.4   4.513000011444092   2017/11/01  12:09:56
-27.7791    22.9941 3   21.4    413.7   3.8450000286102295  2017/11/01  12:10:00
-27.7788    22.994  3   23.1    430.8   3.485999822616577   2017/11/01  12:10:04

Mi Dataset

lat        lon      EventDate   EventTime
-27.7786    22.9939 2017/11/01  12:10:07
-27.7784    22.9939 2017/11/01  12:10:10
-27.7782    22.9939 2017/11/02  12:10:14
-27.778     22.9938 2017/11/02  12:10:17
-27.7777    22.9938 2017/11/02  12:10:21

Linked_df

lat_t   lon_t   y   EventDate   EventTime
-27.7786    22.9939 3   2017/11/01  12:10:07
-27.7784    22.9939 3   2017/11/01  12:10:10

How can the code be optimized?

NB: Open to dask dataframes solutions as well. There are date that are the same. Note the dataset is larger than the example above and is taking over a week to complete its run. The most important condition is the distance needs to be less than 15 meters and the time difference is 10 sec or less. It is also not required to calculate the duration since it is not stored. There may be alternate ways to determine if the duration is less than 10 seconds that may take less computational time.

Answer 1

If you want speed do not use iterrows() if you can avoid it. Vectorization can give you 50-fold or 100-fold improvement in speeds.

This is an example of how to use vectorization on your code:

for unq_date in R_Unique_Dates:
    M=Mi.loc[(pd.to_datetime(Mi['EventDate']) == unq_date) ]
    R=Ri.loc[(pd.to_datetime(Ri['EventDate']) == unq_date) ]

    M['date'] = pd.to_datetime(date.today() +' '+ M['EventTime'])
    R['date'] = pd.to_datetime(date.today() +' '+ R['EventTime'])

    M['duration'] = M['date'] - R['date']
    M.loc[M.duration < 0, 'duration'] =  R['date'] - M['date']
    ...

This way you will avoid using iterrows().

This code might not work out of the box given that we don't have the data you are using, but you should follow this idea: do the operation in the whole dataframe at the same time (vectorization) rather than iterating over it (iterrows()). Loops are bad for performance. This article is great at explaining this concept.

The outer loop for unq_date in R_Unique_Dates: can be expressed as a groupby, but I would recommend starting with the above. Using groupby can be a bit confusing when you are starting.