Pandas groupby object.aggregate具有自定义列表操作功能

Question

I have a csv file as shown below

Hour,L,Dr,Tag,Code,Vge
0,L5,XI,PS,4R,15
0,L3,St,sst,4R,17
5,L5,XI,PS,4R,12
2,L0,St,v2T,4R,11
8,L2,TI,sst,4R,8
12,L5,XI,PS,4R,18
2,L2,St,PS,4R,9
12,L3,XI,sst,4R,16

I execute the following script in my ipython notebook.

In[1]
    import pandas as pd
In[2]
    df = pd.read_csv('/python/concepts/pandas/in.csv')
In[3]    
    df.head(n=9)

Out[1]: 

       Hour   L  Dr  Tag Code  Vge
    0     0  L5  XI   PS   4R   15
    1     0  L3  St  sst   4R   17
    2     5  L5  XI   PS   4R   12
    3     2  L0  St  v2T   4R   11
    4     8  L2  TI  sst   4R    8
    5    12  L5  XI   PS   4R   18
    6     2  L2  St   PS   4R    9
    7    12  L3  XI  sst   4R   16

In[4]
    df.groupby(('Hour'))['Vge'].aggregate(np.sum)



Out[2]:  
     Hour
        0     32
        2     20
        5     12
        8      8
        12    34
        Name: Vge, dtype: int64

Now I write a list manipulation square_list .

In[4]    

    newlist = []
In[5]    
    def square_list(x):
        for item in x:
            newlist.append(item**item)
        return newlist

In [44]: df.groupby(('Hour'))['Vge'].aggregate(square_list)
Out[44]: 
Hour
0     [437893890380859375, -2863221430593058543, 437...
2     [437893890380859375, -2863221430593058543, 437...
5     [437893890380859375, -2863221430593058543, 437...
8     [437893890380859375, -2863221430593058543, 437...
12    [437893890380859375, -2863221430593058543, 437...
Name: Vge, dtype: object

The output looks strange.All I am expecting is the squares of the items in the first output.

If I use

df.groupby(('Hour'))['Vge'].aggregate(lambda x: x ** x)

I get the following error.

ValueError                                Traceback (most recent call last)
/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in agg_series(self, obj, func)
   1632         try:
-> 1633             return self._aggregate_series_fast(obj, func)
   1634         except Exception:

/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in _aggregate_series_fast(self, obj, func)
   1651                                     dummy)
-> 1652         result, counts = grouper.get_result()
   1653         return result, counts

pandas/src/reduce.pyx in pandas.lib.SeriesGrouper.get_result (pandas/lib.c:38634)()

pandas/src/reduce.pyx in pandas.lib.SeriesGrouper.get_result (pandas/lib.c:38503)()

pandas/src/reduce.pyx in pandas.lib._get_result_array (pandas/lib.c:32023)()

ValueError: function does not reduce

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in aggregate(self, func_or_funcs, *args, **kwargs)
   2339             try:
-> 2340                 return self._python_agg_general(func_or_funcs, *args, **kwargs)
   2341             except Exception:

/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in _python_agg_general(self, func, *args, **kwargs)
   1167             try:
-> 1168                 result, counts = self.grouper.agg_series(obj, f)
   1169                 output[name] = self._try_cast(result, obj)

/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in agg_series(self, obj, func)
   1634         except Exception:
-> 1635             return self._aggregate_series_pure_python(obj, func)
   1636 

/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in _aggregate_series_pure_python(self, obj, func)
   1668                         isinstance(res, list)):
-> 1669                     raise ValueError('Function does not reduce')
   1670                 result = np.empty(ngroups, dtype='O')

ValueError: Function does not reduce

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-47-874cf4c23d53> in <module>()
----> 1 df.groupby(('Hour'))['Vge'].aggregate(lambda x : x**x)

/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in aggregate(self, func_or_funcs, *args, **kwargs)
   2340                 return self._python_agg_general(func_or_funcs, *args, **kwargs)
   2341             except Exception:
-> 2342                 result = self._aggregate_named(func_or_funcs, *args, **kwargs)
   2343 
   2344             index = Index(sorted(result), name=self.grouper.names[0])

/Applications/anaconda/lib/python3.5/site-packages/pandas/core/groupby.py in _aggregate_named(self, func, *args, **kwargs)
   2429             output = func(group, *args, **kwargs)
   2430             if isinstance(output, (Series, Index, np.ndarray)):
-> 2431                 raise Exception('Must produce aggregated value')
   2432             result[name] = self._try_cast(output, group)
   2433 

Exception: Must produce aggregated value

Answer 1

Are you reading the error closely? It says the function does not reduce. Please take a few minutes to define what you want correctly. That is the exact problem with your square_list() function as well, it is returning a list, not the sum of the list elements. It does not reduce.

1. If you want plain sum:

    df.groupby('Hour')['Vge'].sum() 

2. If you want to square all elements in the column:

    df['Vge_squared'] = df['Vge']**2 

3. If you want the group sum of squares:

    df.groupby('Hour')['Vge_squared'].sum() 

Or,

def square_list(x):
    x = numpy.array(x)
    return numpy.sum(numpy.multiply(x,x))

df.groupby('Hour')['Vge'].aggregate(square_list)

Or,

def square_list(x):
    for item in x:
        newlist.append(item**item)
    return newlist

df.groupby('Hour')['Vge'].aggregate(square_list).apply(sum)

Hope this helps.

Answer 2

Firstly, the first output is "as expected", as each call to square_list is appending to the global newlist .

Potentially you could create the list on each call:

def square_list(x):
    newlist = []
    for item in x:
        newlist.append(item**item)
    return newlist

In [11]: df.groupby(('Hour'))['Vge'].aggregate(square_list)
Out[11]:
Hour
0     [437893890380859375, -2863221430593058543]
2                      [285311670611, 387420489]
5                                [8916100448256]
8                                     [16777216]
12                      [-497033925936021504, 0]
dtype: object

but I suspect that's not what you want either.

---

The error message is pretty accurate: "Must produce aggregated value". At the moment your lambda isn't returning a single value.

Perhaps you want the sum:

In [21]: df.groupby(('Hour'))['Vge'].aggregate(lambda x: (x ** x).sum())
Out[21]:
Hour
0    -8785478146473916416
2            285699091100
5           8916100448256
8                16777216
12                      0
Name: Vge, dtype: int64

Note: it may be faster to create a dummy column for the squares then a "clean" groupby sum.