如何使用 Python 映射和减少我的字典列表

Question

我有这个字典列表：

[{'topic_id': 1, 'average': 5.0, 'count': 1}, {'topic_id': 1, 'average': 8.0, 'count': 1}, {'topic_id': 2, 'average': 5.0, 'count': 1}]

我想映射和减少（或分组）以获得如下结果：

[
    {
        'topic_id': 1,
        'count': 2,
        'variance': 3.0,
        'global_average': 6.5
        },
    {
        'topic_id': 2,
        'count': 1,
        'variance': 5.0,
        'global_average': 5.0
    }
]

计算方差（最大平均值 - 最小平均值）并对项目数求和的东西。

我已经做了什么：

在我尝试总结改变字典结构的计数，并使键成为 topic_id 并为计数赋值之前，我的结果是：

result = sorted(dict(functools.reduce(operator.add, map(collections.Counter, data))).items(), reverse=True)

这只是第一次尝试。

Answer 1

这是使用itertools.groupby根据topic_id对数据进行分组的尝试：

import itertools

data = [{'topic_id': 1, 'average': 5.0, 'count': 1}, {'topic_id': 1, 'average': 8.0, 'count': 1}, {'topic_id': 2, 'average': 5.0, 'count': 1}]

# groupby
grouper = itertools.groupby(data, key=lambda x: x['topic_id'])

# holder for output
output = []

# iterate over grouper to calculate things
for key, group in grouper:

    # variables for calculations
    count = 0
    maxi = -1
    mini = float('inf')
    total = 0

    # one pass over each dictionary
    for g in group:
        avg = g['average']
        maxi = avg if avg > maxi else maxi
        mini = avg if avg < mini else mini
        total += avg
        count += 1

    # write to output
    output.append({'total_id':key,
                   'count':count,
                   'variance':maxi-mini,
                   'global_average':total/count})

给出这个output ：

[{'total_id': 1, 'count': 2, 'variance': 3.0, 'global_average': 6.5},
 {'total_id': 2, 'count': 1, 'variance': 0.0, 'global_average': 5.0}]

请注意，第二组的'variance'在这里是0.0而不是5.0 ； 这与您的预期输出不同，但我猜这就是您想要的？

Answer 2

您可以通过一些推导式、一个map和内置statistics模块中的mean函数来实现这一点。

from statistics import mean
data = [
    {
        'topic_id': 1, 
        'average': 5.0, 
        'count': 1
    }, {
        'topic_id': 1, 
        'average': 8.0, 
        'count': 1
    }, {
        'topic_id': 2, 
        'average': 5.0, 
        'count': 1
    }
]
# a set of unique topic_id's
keys = set(i['topic_id'] for i in data)
# a list of list of averages for each topic_id
averages = [[i['average'] for i in data if i['topic_id'] == j] for j in keys]
# a map of tuples of (counts, variances, averages) for each topic_id
stats = map(lambda x: (len(x), max(x) - min(x), mean(x)), averages)
# finally reconstruct it back into a list
result = [
    {
        'topic_id': key, 
        'count': count, 
        'variance': variance, 
        'global_average': average
    } for key, (count, variance, average) in zip(keys, stats)
]
print(result)

退货

[{'topic_id': 1, 'count': 2, 'variance': 3.0, 'global_average': 6.5}, {'topic_id': 2, 'count': 1, 'variance': 0.0, 'global_average': 5.0}]

Answer 3

如果您愿意使用熊猫，这似乎是一个合适的用例：

import pandas as pd

data = [{'topic_id': 1, 'average': 5.0, 'count': 1}, {'topic_id': 1, 'average': 8.0, 'count': 1}, {'topic_id': 2, 'average': 5.0, 'count': 1}]

# move to dataframe
df = pd.DataFrame(data)

# groupby and get all desired metrics
grouped = df.groupby('topic_id')['average'].describe()
grouped['variance'] = grouped['max'] - grouped['min']

# rename columns and remove unneeded ones
grouped = grouped.reset_index().loc[:, ['topic_id', 'count', 'mean', 'variance']].rename({'mean':'global_average'}, axis=1)

# back to list of dicts
output = grouped.to_dict('records')

output是：

[{'topic_id': 1, 'count': 2.0, 'global_average': 6.5, 'variance': 3.0},
 {'topic_id': 2, 'count': 1.0, 'global_average': 5.0, 'variance': 0.0}]

Answer 4

您也可以尝试像这样使用 Pandas 数据框的 agg 功能

import pandas as pd

f = pd.DataFrame(d).set_index('topic_id')

def var(x):
    return x.max() - x.min()

out = f.groupby(level=0).agg(count=('count', 'sum'),
        global_average=('average', 'mean'),
        variance=('average', var))