How to move the datetime to end of day, week or hour

Question

Here is the code that move all the times to the end of month:

import numpy as np
import pandas as pd

times = np.array([
       '2013-07-22T02:10:32.000000000+0900',
       '2013-07-22T01:11:13.000000000+0900',
       '2013-07-21T23:23:32.000000000+0900',
       '2013-07-21T05:59:21.000000000+0900',
       '2013-07-21T05:57:30.000000000+0900',
       '2013-07-21T05:44:27.000000000+0900',
       '2013-07-20T10:45:17.000000000+0900',
       '2013-07-20T10:36:53.000000000+0900',
       '2013-07-20T09:57:46.000000000+0900',
       '2013-07-20T09:57:06.000000000+0900',
       '2013-07-20T09:30:57.000000000+0900',
       '2013-07-20T08:20:27.000000000+0900',], dtype='datetime64[ns]')

dti = pd.DatetimeIndex(times)
dti.shift(1, "M").values

The result is:

array(['2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900',
       '2013-07-31T09:00:00.000000000+0900'], dtype='datetime64[ns]')

but how to move all the times to the end of the hour, day or week?

Answer 1

The best method i found for this is by to_period & to_timestamp :

In [39]:

dti.to_period("W-SAT").to_timestamp(how="end").values

Out[39]:

array(['2013-07-27T09:00:00.000000000+0900',
       '2013-07-27T09:00:00.000000000+0900',
       '2013-07-27T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900',
       '2013-07-20T09:00:00.000000000+0900'], dtype='datetime64[ns]')

In [40]:

dti.to_period("H").to_timestamp(how="end").values

Out[40]:

array(['2013-07-22T02:59:59.000000000+0900',
       '2013-07-22T01:59:59.000000000+0900',
       '2013-07-21T23:59:59.000000000+0900',
       '2013-07-21T05:59:59.000000000+0900',
       '2013-07-21T05:59:59.000000000+0900',
       '2013-07-21T05:59:59.000000000+0900',
       '2013-07-20T10:59:59.000000000+0900',
       '2013-07-20T10:59:59.000000000+0900',
       '2013-07-20T09:59:59.000000000+0900',
       '2013-07-20T09:59:59.000000000+0900',
       '2013-07-20T09:59:59.000000000+0900',
       '2013-07-20T08:59:59.000000000+0900'], dtype='datetime64[ns]')

Answer 2

I agree with Andy; that can't be the intended behavior of shift . A cleaner way to shift times to the end of the month is this:

from pandas.tseries.offsets import MonthEnd
times = Series(times)
times.map(lambda x: x + MonthEnd())

But there is no such thing as HourEnd, DayEnd, or WeekEnd. For those cases, how about following this pattern?

from pandas.tseries.offsets import Second, Minute, Hour, Day

times.map(lambda x: x + Minute(59-x.minute) + Second(59-x.second))

times.map(lambda x: x + Hour(23-x.hour) + Minute(59-x.minute) + Second(59-x.second))

times.map(lambda x: x + Day(6-x.weekday()) + Hour(23-x.hour) + \
          Minute(59-x.minute) + Second(59-x.second))

If you want the last day of the week but not necessarily the last second of that day, then the expression is obviously simpler.