在大型 csv 文件上将每分钟数据的时间序列聚合到每小时

Question

Hello I have a question about how to aggregate per minute data to an hourly level for each road segment. The data should be grouped by hour and the road segment ID. Would this be possible to do on 15gb+ csv as I have only filtered only the relevant road segments to reduce the size to 1-2GB?

The data set is something like this

             DateTime              SegmentID    Speed
    2019-10-08T01:00:00+01:00          1          39
    2019-10-08T01:00:01+01:00          1          39

Answer 1

You can use the resample() pandas function. Then it looks like you'd want to get the sum() or mean() of the other columns?

df.resample('H').mean()

If your DateTime column is not your index you will have to do:

df.resample('H', on='DateTime').mean()

You can use other aggregations instead of mean() or sum() depending on what you are trying to achieve.