使用熊猫有效地计算大型数据帧的每个时间仓的值

Question

I have multiple large dataframes (ca. 3GB csv files with ca. 150 million rows each) that contains Unix-style timestamps and random-generated observation ids. Each observation can/will occur multiple times at different times. They look like this:

    time_utc    obs_id
0   1564617600  aabthssv
1   1564617601  vvvx7ths
2   1564618501  optnhfsa
3   1564619678  aabthssv
4   1564619998  abtzsnwe
         ...

I now want to in order to analyse the time development of observations get a data frame that contains columns for each observation id and rows for a time bin that can be changed, eg 1 hour, like this:

time_bin aabthssv vvvx7ths optnhfsa  ...
1               1        1        1
2               1        0        0
               ...

I have tried to do this by creating a numpy array of timestamp start points and then adding value_counts for a selection of all rows in that bin to a new, empty dataframe. This runs into MemoryError. I have tried pre-cleaning more, but even reducing the size of the raw data by a third (so 2GB, 100 million rows) still has Memory Errors occuring.

SLICE_SIZE = 3600 # example value of 1h
slice_startpoints = np.arange(START_TIME, END_TIME+1-SLICE_SIZE, SLICE_SIZE)
agg_df = pd.DataFrame()

for timeslice in slice_startpoints:
        temp_slice = raw_data[raw_data['time_utc'].between(timeslice, timeslice + SLICE_SIZE)]
        temp_counts = temp_slice['obs_id'].value_counts()
        agg_df = agg_df.append(temp_counts)
        temp_index = raw_data[raw_data['time_utc'].between(timeslice, timeslice + SLICE_SIZE)].index
        raw_data.drop(temp_index, inplace=True)

Is there a way to do this more efficiently or rather so that it works at all?

Edit: I found my efficient way to do it based on the suggestion to use crosstab. The file size did not need to be reduced. Using the following code resulted in exactly the result I was looking for.

df['binned'] = pd.cut(df['time_utc'], bins=slice_startpoints, include_lowest=True, labels=slice_startpoints[1:])
df.groupby('binned')['obs_id'].value_counts().unstack().fillna(0)

Answer 1

You can try cut with crosstab :

slice_startpoints = np.arange(START_TIME, END_TIME+SLICE_SIZE, SLICE_SIZE)
print (slice_startpoints)

df['binned'] = pd.cut(df['time_utc'], 
                      bins=slice_startpoints, 
                      include_lowest=True,
                      labels=slice_startpoints[1:])

df = pd.crosstab(df['binned'], df['obs_id'])

Answer 2

You could read in a large .csv with the 'chunk' iterator, and then perform the calculation on the chunk instead of the the entire .csv file. The chunksize defines the number of rows in a single chunk. That way, you have a good handle to control the memory usage. The downside will be that you will have to add some logic that will merge the results of the chunks.

import pandas as pd
df_chunk = pd.read_csv('file.csv', chunksize=1000)
for chunk in df_chunk:
    print(chunk)