Pandas：如何在 groupby() 操作后有效地 diff()？

Question

I do have a large dataset (around 8 million rows x 25 columns) in Pandas and I am struggling to use the diff() function in a performant manner on a subset of the data.

Here is how my dataset looks like:

                   prec   type
location_id hours             
135         78     12.0      A
            79     14.0      A
            80     14.3      A
            81     15.0      A
            82     15.0      A
            83     15.0      A
            84     15.5      A

• I have a multi-index on [location_id, hours]. I have around 60k locations and 140 hours for each location (making up the 8 million rows).
• The rest of the data is numeric (float) or categorical. I have only included 2 columns here, normally there are around 20 columns.
• What I am willing to do is to apply the diff() function for each location on the prec column. The original dataset piles up the prec numbers; by applying diff() I will get the appropriate prec value for each hour.
• With these in mind, I have implemented the following algorithm in Pandas:

# Filter the data first
df_filtered = df_data[df_data.type == "A"] # only work on locations with 'A' type
df_filtered = df_filtered.query('hours > 0 & hours <= 120') # only work on certain hours

# Apply the diff()
for location_id, data_of_location in df_filtered.groupby(level="location_id"):
    df_data.loc[data_of_location.index, "prec"] = data_of_location.prec.diff().replace(np.nan, 0.0)
del df_filtered

• This works really well functionally, however the performance and the memory consumption is horrible. It is taking around 30 minutes on my dataset and that is currently not acceptable. The existence of the for loop is an indicator that this could be handled better.
• Is there a better/faster way to implement this?
• Also, the overall memory consumption of the Python script is sky-rocketing during this operation; it grows around 300%! The memory consumed by the main df_data data frame doesn't change but the overall process memory consumption rises.

Answer 1

With the input from @Quang Hoang and @Ben. T, I figured out a solution that is pretty fast but still consumes a lot of memory.

# Filter the data first
df_filtered = df_data[df_data.type == "A"] # only work on locations with 'A' type
df_filtered = df_filtered.query('hours > 0 & hours <= 120') # only work on certain hours

# Apply the diff()
df_diffed = df_data.groupby(level="location_id").prec.diff().replace(np.nan, 0.0)
df_data[df_diffed.index, "prec"] = df_diffed
del df_diffed
del df_filtered

I am guessing 2 things can be done to improve memory usage:

• df_filtered seems like a copy of the data; that should increase the memory a lot.
• df_diffed is also a copy.

The memory usage is very intensive while computing these two variables. I am not sure if there is any in-place way to execute such operations.