重新采样/重新索引传感器数据

Question

I want to do some data processing to sensor data (about 300 different sensors). This is an example of the raw data from a temperature sensor:

 "2018-06-30T13:17:05.986Z" 30.5
 "2018-06-30T13:12:05.984Z" 30.3
 "2018-06-30T13:07:05.934Z" 29.5
 "2018-06-30T13:02:05.873Z" 30.3
 "2018-06-30T12:57:05.904Z" 30

I want to resample the data to smooth datetimes:

13:00:00
13:05:00
13:10:00
...

I have written some code that works, but is incredibly slow when used on bigger files. My code just upsamples all the data to 1 sec via linear interpolation. and downsamples afterwards to the requested frequency.

Is there a faster method to achieve this?

EDIT: sensor data is written into a database and my code loads data from an arbitrary time intervall from the database

EDIT2: My working code

upsampled = dataframe.resample('1S').asfreq()
upsampled = upsampled.interpolate(method=method, limit=limitT) # ffill or bfill for some sensors 
resampled = upsampled.astype(float).resample(str(sampling_time) + 'S').mean() # for temperature 
resampled = upsampled.astype(float).resample(str(sampling_time) + 'S').asfreq() # for everything else

Answer 1

You can first set the index for the dataframe as the column with timestamps, and then use resample() method to bring it to every 1sec or every 5min interval data.

For example:

temp_df = pd.read_csv('temp.csv',header=None)
temp_df.columns = ['Timestamps','TEMP']
temp_df = temp_df.set_index('Timestamps') #set the timestamp column as index
temp_re_df = temp_df.TEMP.resample('5T').mean()

You can set the period as argument to the resample() ie T - min , S - sec , M - month, H - hour etc. and also apply a function like mean() or max() or min() to consider the down-sampling method.

PS : This is given that that your timestamp are in datetime format of pandas. Else use pd.to_datetime(temp_df['Timestamps'],unit='s') to convert to datetime index column