How to calculate mean and variance from pandas datetime object?

Question

How do I calculate summary statistics (mean and standard deviation) for python datetime objects in the form YYYY-MM-DD? I want to do this for different groups of datetime obejcts which have different IDs.

Here's what the data look like:

import datetime as dt

df = pd.DataFrame({
'Date': [dt.date(2017,9,1),dt.date(2017,9,21),dt.date(2017,9,14),
    dt.date(2017,11,7),dt.date(2017,8,1),dt.date(2017,12,21),
    dt.date(2017,12,14),dt.date(2017,10,1),dt.date(2017,10,1)],
'ID': [1,2,3,3,2,1,2,3,2],
})

    Date        ID
    2017-09-01  1
    2017-11-01  2
    2017-09-01  3
    2017-11-07  3
    2017-08-01  2
    2017-11-01  1
    2017-12-01  2
    2017-10-01  3
    2017-10-01  2

And I want a result that looks like:

ID_1_mean  ID_1_sd  ID_2_mean   ID_2_sd ...
YYYY-MM-DD int      YYYY-MM-DD  int ...

where YYYY-MM-DD is the mean from the dates in group 1 and int is the mean in group 1, repeated for all the groups.

Answer 1

Here's a somewhat clunky workaround:

1. Convert datetime.date to pandas.Timestamp with pd.to_datetime()
2. Convert pandas.Timestamp to integer with .astype(int)
3. Compute mean and std of these integers
4. Convert mean to pandas.Timestamp
5. Convert std to pandas.Timedelta

Setup:

df = pd.DataFrame({
'Date': [dt.date(2017,9,1),dt.date(2017,9,21),dt.date(2017,9,14),
    dt.date(2017,11,7),dt.date(2017,8,1),dt.date(2017,12,21),
    dt.date(2017,12,14),dt.date(2017,10,1),dt.date(2017,10,1)],
'ID': [1,2,3,3,2,1,2,3,2],
})

Solution:

df['Date_int'] = pd.to_datetime(df['Date']).astype(int)
res = df.groupby('ID').agg(['mean', 'std'])
res.columns = ['_'.join(c) for c in res.columns.values]

res['Date_mean'] = pd.to_datetime(res['Date_int_mean'])
res['Date_std'] = pd.to_timedelta(res['Date_int_std'])

res = res[['Date_mean', 'Date_std']]
res

             Date_mean                Date_std
ID                                            
1  2017-10-26 12:00:00 78 days 11:43:56.874291
2  2017-10-01 18:00:00 55 days 15:53:10.401720
3  2017-10-07 16:00:00 27 days 14:38:57.222514