获取时间戳在不规则时间间隔内的行熊猫（时间序列）

Question

Let's say I have a dataframe like this:

>>> i = pd.to_datetime(np.random.randint(time.time(), time.time()+10000, 15), unit='ms').sort_values()
>>> df = pd.DataFrame({'A':range(15),'B':range(10,40,2),'C':range(10,55,3)},index = i)
>>> df
                          A   B   C
1970-01-19 05:31:36.629   0  10  10
1970-01-19 05:31:36.710   1  12  13
1970-01-19 05:31:37.779   2  14  16
1970-01-19 05:31:38.761   3  16  19
1970-01-19 05:31:39.520   4  18  22
1970-01-19 05:31:39.852   5  20  25
1970-01-19 05:31:39.994   6  22  28
1970-01-19 05:31:41.370   7  24  31
1970-01-19 05:31:41.667   8  26  34
1970-01-19 05:31:42.515   9  28  37
1970-01-19 05:31:42.941  10  30  40
1970-01-19 05:31:43.037  11  32  43
1970-01-19 05:31:43.253  12  34  46
1970-01-19 05:31:43.333  13  36  49
1970-01-19 05:31:44.135  14  38  52

What I want is:

                          A   B   C
1970-01-19 05:31:37.779   2.0  14.0  16.0   #last value within 2000 milli-sec interval from 05:31:36
1970-01-19 05:31:38.761   3.0  16.0  19.0      ##last value from the ^ value within 1000 msec interval
1970-01-19 05:31:39.994   6.0  22.0  28.0   #last value within 2000 milli-sec interval from 05:31:38
1970-01-19 05:31:39.994   6.0  22.0  28.0     *##last value from the ^ value within 1000 msec interval
1970-01-19 05:31:41.667   8.0  26.0  34.0   #last value within 2000 milli-sec interval from 05:31:40
1970-01-19 05:31:42.515   9.0  28.0  37.0      ##last value from the ^ value within 1000 msec interval
1970-01-19 05:31:43.333  13.0  36.0  49.0   #last value within 2000 milli-sec interval from 05:31:42
1970-01-19 05:31:44.135  14.0  38.0  52.0      ##last value from the ^ value within 1000 msec interval

I can achieve the rows marked with # s with this code:

>>> df.resample('2000ms').ffill().dropna(axis=0)
                        A     B     C
1970-01-19 05:31:38   2.0  14.0  16.0
1970-01-19 05:31:40   6.0  22.0  28.0
1970-01-19 05:31:42   8.0  26.0  34.0
1970-01-19 05:31:44  13.0  36.0  49.0

# note I do not care about how the timestamps are getting printed, I just want the correct values.

I can't find a solution with pandas that will give me the desired output. I can do this with two dataframes, one sampled at 2000ms and another at 1000ms and then loop probably, and inserting accordingly.

The problem is, the actual size of my data is really large, with over 4000000 rows and 52 columns. If it is possible to avoid two dfs, or loops, I would definitely want to take that.

NOTE : The * marked row gets repeated, as there are no data within 1000ms time interval from the last value, so the last seen value is repeated. The same should happen for 2000ms time intervals as well.

If possible please show me a way... Thanks!

EDIT : Edited as per John Zwinck's comment :

import datetime
def last_time(time):
    time = str(time)
    start_time = datetime.datetime.strptime(time[11:],'%H:%M:%S.%f')
    end_time = start_time + datetime.timedelta(microseconds=1000000)
    tempdf = df.between_time(*pd.to_datetime([str(start_time),str(end_time)]).time).iloc[-1]
    return tempdf
df['timestamp'] = df.index
df2 = df.resample('2000ms').ffill().dropna(axis=0)
df3 = df2.apply(lambda x:last_time(x['timestamp']), axis = 1)

pd.concat([df2, df3]).sort_index(kind='merge')

This gives:

                        A     B     C               timestamp
1970-01-19 05:31:38   2.0  14.0  16.0 1970-01-19 05:31:37.779
1970-01-19 05:31:38   3.0  16.0  19.0 1970-01-19 05:31:38.761
1970-01-19 05:31:40   6.0  22.0  28.0 1970-01-19 05:31:39.994
1970-01-19 05:31:40   6.0  22.0  28.0 1970-01-19 05:31:39.994
1970-01-19 05:31:42   8.0  26.0  34.0 1970-01-19 05:31:41.667
1970-01-19 05:31:42   9.0  28.0  37.0 1970-01-19 05:31:42.515
1970-01-19 05:31:44  13.0  36.0  49.0 1970-01-19 05:31:43.333
1970-01-19 05:31:44  14.0  38.0  52.0 1970-01-19 05:31:44.135

Which is okay, except the apply part takes really long time!

---

For easier copy:

,A,B,C
1970-01-19 05:31:36.629,0,10,10
1970-01-19 05:31:36.710,1,12,13
1970-01-19 05:31:37.779,2,14,16
1970-01-19 05:31:38.761,3,16,19
1970-01-19 05:31:39.520,4,18,22
1970-01-19 05:31:39.852,5,20,25
1970-01-19 05:31:39.994,6,22,28
1970-01-19 05:31:41.370,7,24,31
1970-01-19 05:31:41.667,8,26,34
1970-01-19 05:31:42.515,9,28,37
1970-01-19 05:31:42.941,10,30,40
1970-01-19 05:31:43.037,11,32,43
1970-01-19 05:31:43.253,12,34,46
1970-01-19 05:31:43.333,13,36,49
1970-01-19 05:31:44.135,14,38,52

Answer 1

The slow part of your existing code is the creation of df3 , so I'll optimize that.

First, note that your last_time(x) function looks for the last record within the time range from x to x + 1 second.

Instead of using a loop, we can start by offsetting the time in the entire vector:

end_times = df2.timestamp + datetime.timedelta(microseconds=1000000)

Then we can use numpy.searchsorted() (a highly underrated function!) to search for those times in df :

idx = np.searchsorted(df.timestamp, end_times)

Incidentally, df.timestamp.searchsorted(end_times) does the same thing.

Finally, note that each of those generated indexes is one after what we want (we don't want the values 1 second after, we want the one just before that):

df3a = df.iloc[idx - 1]

This gives the same result as your df3 except the index is not rounded down, so just replace it:

df3a.index = df2.index

This is exactly the same as your df3 , but calculated much more quickly.