How can I count the number of elements given to a column sum from a list of dictionaries quickly?

Question

Whew! That title is a mouthful.

I have a list of dictionaries, and from it, I'm trying to extract the following:

for code in codes:
    for type in types:
        for color in colors:
            count  = sum(1               for d in data if d.get('code') == code and d.get('type') == type and d.get('color') == color)
            amount = sum(d.get('amount') for d in data if d.get('code') == code and d.get('type') == type and d.get('color') == color)
            cost   = sum(d.get('cost')   for d in data if d.get('code') == code and d.get('type') == type and d.get('color') == color)

I know I could change the last two lines to:

tally = {k: sum(d.get(k) for d in data if d.get('code') == code and d.get('type') == type and d.get('color') == color) for k in ('amount', 'cost')}

but is there away to include the count in the above line as well?

Answer 1

您可以在此处使用条件表达式 ：

tally = {k: sum(d.get(k) if k != 'count' else 1 for d in data if d.get('code') == code and d.get('type') == type and d.get('color') == color) for k in ('count', 'amount', 'cost')}

Answer 2

Instead of trawling through d for each (code,type,color) combination, I would pass through it once and store the results:

from collections import defaultdict

counts  = defaultdict(int)
amounts = defaultdict(float)
costs   = defaultdict(float)

for d in data:
    code  = d.get('code')
    type  = d.get('type')
    color = d.get('color')
    if code in codes and type in types and color in colors:
        key = (code, type, color)
        counts [key] += 1
        amounts[key] += d.get('amount')
        costs  [key] += d.get('cost')

For maximum speed, codes, types, and colors should each be either a set or the keys of a dict.

Answer 3

Given the triply-nested for loops here, I think it would be worth your while to index the list of dictionaries:

from collections import defaultdict
data_entries = defaultdict(list)
for d in data:
    data_entries[(d['code'], d['type'], d['color'])].append(d)
for code in codes:
    for type in types:
        for color in colors:
            entries = data_entries[(code, type, color)]
            count = len(entries)
            amount = sum(d.get('amount', 0) for d in entries)
            cost = sum(d.get('cost', 0) for d in entries)

Then you could also use the conditional expression from Ashwini Chaudhary's answer, but it should be significantly faster when you only have to sum over the actually matching entries. tally['count'] = len(entries) should be faster than the dict comprehension, but it does require slightly more code.