使用 petl 使用 python 按多个字段分组

Question

I am trying to group by multiple fields a table that has already been read in using petl (that is, it's a petl type). I am trying with the following piece of code:

import petl as etl
from collections import OrderedDict

csv_ea_flood_def = r'............'   #Path to csv table
ea_flood_def = etl.fromcsv(csv_ea_flood_def)

aggreg = OrderedDict()
aggreg['count'] = len
aggreg['sum_length'] = 'length', sum
EA_Defence_Types = etl.aggregate(ea_flood_def, key=('prot_type', 'asset_type', 'description'), aggregation=aggreg)

This piece of code does not throw any error. However, the resulting table seems to be empty or corrupt in some way:

print(EA_Defence_Types[0]) will throw:

('prot_type', 'asset_type', 'description', 'count', 'sum_length')  #which is fine

print(EA_Defence_Types[1]) will throw:

Traceback (most recent call last):
  File "def_failure_prob.py", line 514, in <module>
    main()
  File "def_failure_prob.py", line 504, in main
    def_fail_prob(con, cursor)
  File "def_failure_prob.py", line 111, in def_fail_prob
    print(EA_Defence_Types[1])
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\util\base.py", line 172, in __getitem__
    return super(Table, self).__getitem__(item)
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\util\base.py", line 33, in __getitem__
    return next(islice(self, item, item+1))
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\transform\reductions.py", line 308, in itermultiaggregate
    for k, rows in rowgroupby(it, key):
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\util\base.py", line 715, in <genexpr>
    return ((k.inner, vals) for (k, vals) in git)
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\util\base.py", line 699, in <genexpr>
    it = (Record(row, flds) for row in it)
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\transform\sorts.py", line 354, in _iternocache
    for row in _mergesorted(getkey, reverse, *chunkiters):
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\transform\sorts.py", line 164, in _heapqmergesorted
    for element in heapq.merge(*keyed_iterables):
  File "C:\ProgramData\Anaconda3\envs\env1\lib\heapq.py", line 350, in merge
    h_append([next(), order * direction, next])
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\transform\sorts.py", line 162, in <genexpr>
    keyed_iterables = [(_Keyed(key(obj), obj) for obj in iterable)
  File "C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\petl\transform\sorts.py", line 124, in _iterchunk
    yield pickle.load(f)
TypeError: __new__() missing 1 required positional argument: 'flds'

I have no idea what that error could mean.

Answer 1

I found the problem. The filed I was using to calculate 'sum' was not numeric (I just had to convert from string to float).