How do I calculate the distance between a row and the row immediately before it for each different value of a third column?

Question

I have a dataframe with devices, dates, and lat/lon columns like this:

e5c0e3a5    2019-09-23 00:25:48 -44.132 -30.369
e5c0e3a5    2019-09-23 00:30:48 -43.437 -30.633
...
a5c0d8b8    2019-09-23 03:20:48 -30.132 -40.369
a5c0d8b8    2019-09-23 03:50:12 -30.437 -41.633

the records are sorted by user and by time. I need to measure the distance travelled by each user at time t, and time t+1 (or, to avoid the first nan, t and t-1, starting at row 2).

I'm using the from geopy.distance import geodesic function to calculate the distance, and would like resulting dataframe of the style:

e5c0e3a5    2019-09-23 00:25:48 20
a5c0d8b8    2019-09-23 03:50:12 50
...

where I calculated the distance in kms to be 20 by taking row 2 and measuring the distance wrt row 1.

In more general terms, how do I carry out an operation ( geodesic ) for each different device between a row and the row immediately before it?

Answer 1

• I did try a vectorized approach:
  • geodesic(df[['long', 'lat']].to_numpy(), df[['s_long', 's_lat']].to_numpy())
  • However, geodesic does not work with an array.

Distance between each sequentially shifted row

• Using pandas.Series.shift and .apply

import pandas as pd
from geopy.distance import geodesic

# set up data and dataframe; extra data has been added
data = {'code': ['e5c0e3a5', 'e5c0e3a5', 'e5c0e3a5', 'a5c0d8b8', 'a5c0d8b8', 'a5c0d8b8'],
        'datetime': ['2019-09-23 00:25:48', '2019-09-23 00:30:48', '2019-09-23 00:35:48', '2019-09-23 03:20:48', '2019-09-23 03:50:12', '2019-09-23 04:00:12'],
        'long': [-44.132, -43.437, -40.654, -30.132, -30.437, -30.000],
        'lat': [-30.369, -30.633, -29.00, -40.369, -41.633, -43.345]}

df = pd.DataFrame(data)

# sort the dataframe by code and datetime
df = df.sort_values(['code', 'datetime']).reset_index(drop=True)

# # add a shifted columns
df[['s_long', 's_lat']] = df[['long', 'lat']].shift(-1)

# # drop na; the first shifted row will be nan, which won't work with geodesic
df.dropna(inplace=True)

# # apply geodesic to calculate distance between each sequentially shifted row
df['distance_miles'] = df[['long', 'lat', 's_long', 's_lat']].apply(lambda x: geodesic((x[0], x[1]), (x[2], x[3])).miles, axis=1)

# display(df)
       code             datetime    long     lat  s_long   s_lat  distance_miles
0  a5c0d8b8  2019-09-23 03:20:48 -30.132 -40.369 -30.437 -41.633        78.43026
1  a5c0d8b8  2019-09-23 03:50:12 -30.437 -41.633 -30.000 -43.345       106.74601
2  a5c0d8b8  2019-09-23 04:00:12 -30.000 -43.345 -44.132 -30.369      1206.65789
3  e5c0e3a5  2019-09-23 00:25:48 -44.132 -30.369 -43.437 -30.633        49.76606
4  e5c0e3a5  2019-09-23 00:30:48 -43.437 -30.633 -40.654 -29.000       209.63396

Distances only within code groups

• .groupby 'code' and .GroupBy.apply the function, get_distance .
• The code inside the function is the same as the previous code, except it's being applied group-wise.

def get_distance(d: pd.DataFrame) -> pd.DataFrame:
    v = d.copy()  # otherwise, working on d will do an inplace update to df, which will cause unexpected/undesired results.
    v.drop(columns=['code'], inplace=True)  # code will be in the index, so a code column is not needed
    v[['s_long', 's_lat']] = v[['long', 'lat']].shift(-1)
    v.dropna(inplace=True)
    v['dist_miles'] = v[['long', 'lat', 's_long', 's_lat']].apply(lambda x: geodesic((x['long'], x['lat']), (x['s_long'], x['s_lat'])).miles, axis=1)
    return v


# set up data and dataframe; extra data has been added
data = {'code': ['e5c0e3a5', 'e5c0e3a5', 'e5c0e3a5', 'a5c0d8b8', 'a5c0d8b8', 'a5c0d8b8'],
        'datetime': ['2019-09-23 00:25:48', '2019-09-23 00:30:48', '2019-09-23 00:35:48', '2019-09-23 03:20:48', '2019-09-23 03:50:12', '2019-09-23 04:00:12'],
        'long': [-44.132, -43.437, -40.654, -30.132, -30.437, -30.000],
        'lat': [-30.369, -30.633, -29.00, -40.369, -41.633, -43.345]}

df = pd.DataFrame(data)

# sort the dataframe by code and datetime
df = df.sort_values(['code', 'datetime']).reset_index(drop=True)

# apply the function to the groups
test = df.groupby('code').apply(get_distance)

# display(test)
                       datetime    long     lat  s_long   s_lat  dist_miles
code                                                                       
a5c0d8b8 0  2019-09-23 03:20:48 -30.132 -40.369 -30.437 -41.633    78.43026
         1  2019-09-23 03:50:12 -30.437 -41.633 -30.000 -43.345   106.74601
e5c0e3a5 3  2019-09-23 00:25:48 -44.132 -30.369 -43.437 -30.633    49.76606
         4  2019-09-23 00:30:48 -43.437 -30.633 -40.654 -29.000   209.63396