重复提取熊猫中的行组的有效方法

Question

I have a pandas DataFrame that looks in essence like the following:

Group   Date    Value   etc.
1       01/01   10
1       05/01   10
1       08/01   5
1       15/01   5
1       18/01   2
1       21/01   10
...
2       02/01   3
2       15/01   4
2       25/01   1
...
3       01/01   6
....

I would like extract each Group to a separate pandas dataframe containing all the rows in that group (eg to a dictionary with keys 1,2,3 etc). The obvious way to do this is using a slice (like df[df.Group == 1] ) looping through the groups.

However, with a very quite large data set (700k rows, with 30k groups), the slice technique is quite slow because the entire 700k transactions must be accessed for each of the 30k groups.

Any suggestions for a faster method, where each of the 700k rows only has to be accessed once to perform the groupings? Thanks!

Answer 1

I don't know why you'd want a separate df for each group, I'd just groupby on the 'group' and use the groups attribute to index back into the orig df, or use get_group :

In [79]:
groups = df.groupby('Group')
groups.groups

Out[79]:
{1: [0, 1, 2, 3, 4, 5], 2: [6, 7, 8], 3: [9]}

In [81]:    
groups.get_group(1)

Out[81]:
   Group   Date  Value
0      1  01/01     10
1      1  05/01     10
2      1  08/01      5
3      1  15/01      5
4      1  18/01      2
5      1  21/01     10

In [82]:    
df.loc[groups.groups[1]]

Out[82]:
   Group   Date  Value
0      1  01/01     10
1      1  05/01     10
2      1  08/01      5
3      1  15/01      5
4      1  18/01      2
5      1  21/01     10

Answer 2

You could use groupby on the Group column. This will get you all groups and you will be able to process each group with a function -

df.groupby('Group').<apply function here>

For example -

In [13]: df
Out[13]: 
    Group   Date  Value
0       1  01/01     10
1       1  05/01     10
2       1  08/01      5
3       1  15/01      5
4       1  18/01      2
5       1  21/01     10
6       2  15/01      5
7       2  18/01      2
8       1  21/01     10
9       1  15/01      5
10      5  18/01      2
11      5  21/01     10

In [14]: df.groupby('Group').groups
Out[14]: {1: [0, 1, 2, 3, 4, 5, 8, 9], 2: [6, 7], 5: [10, 11]}

In [15]: grp = df.groupby('Group')

This gets you the Groups 1:

In [16]: grp.get_group(1)
Out[16]: 
   Group   Date  Value
0      1  01/01     10
1      1  05/01     10
2      1  08/01      5
3      1  15/01      5
4      1  18/01      2
5      1  21/01     10
8      1  21/01     10
9      1  15/01      5

The documentation here will help you further - http://pandas.pydata.org/pandas-docs/dev/groupby.html