有什么pythonic方式可以清除此词典列表？

Question

Hello and thanks for your help. I have a list of dictionaries that looks like this:

list_balls = [{'id': '803371', 'is_used': False, 'source': 'store', 'air': 0.9},
{'id': '803371', 'is_used': False, 'source': 'donation', 'air': 0.20},
{'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75},
{'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}]

I need to clean this list leaving unique list of dictionaries. If there is two entries or more with the same Id, I need to pick the one with the highest value on air. If they have equal values on air and ids, I need to leave the one where source == 'store'. Therefore, the result in this case would be

list_balls = [{'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75},
{'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}]

I tried the following code to flag as keep = False for the ones that need to be taken out but it only works when there is two duplicates:

for i in range(0, len(list_balls )):
    if len(list_balls ) > 1:
        #print(list_balls [i])
        for j in range(1, len(list_balls )):
            if (list_balls [i]['id'] == list_balls [j]['id']):
                if (list_balls [i]['air'] > list_balls [j]['air']):
                    list_balls [i]['keep'] = True
                    list_balls [j]['keep'] = False
print(list_pns)

I assume this double for loop is not the most efficient way to do this either so any other ideas are welcome. Thanks for your help

Answer 1

Using itertools.groupby

Ex:

from itertools import groupby
list_balls = [{'source': 'store', 'air': 0.9, 'id': '803371', 'is_used': False}, {'source': 'donation', 'air': 0.2, 'id': '803371', 'is_used': False}, {'source': 'donation', 'air': 0.75, 'id': '30042', 'is_used': False}, {'source': 'store', 'air': 1, 'id': '803371', 'is_used': False}]


#result = [max(list(v), key=lambda x: x["air"]) for k, v in groupby(sorted(list_balls, key=lambda x: x["id"]), lambda x: x["id"])]
result = [max(list(v), key=lambda x: (x["air"], x["source"] == "store")) for k, v in groupby(sorted(list_balls, key=lambda x: x["id"]), lambda x: x["id"])]
print(result)

Output:

[{'air': 0.75, 'id': '30042', 'is_used': False, 'source': 'donation'},
 {'air': 1, 'id': '803371', 'is_used': False, 'source': 'store'}]

Answer 2

Simply with something like this :

list_balls = [{'id': '803371', 'is_used': False, 'source': 'store', 'air': 0.9},
{'id': '803371', 'is_used': False, 'source': 'donation', 'air': 0.20},
{'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75},
{'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}]

result = {}

for e in list_balls:
    if e['id'] not in result or (
          (e['air'], e['source'] == 'store') > 
          (result[e['id']]['air'], result[e['id']]['source'] =='store')
        ):
        result[e['id']] = e

result_list = list(result.values())

print(result_list)

Displays

[{'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}, {'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75}]

You can compare directly tuples to compare on multiple criterion. Notice that True is always > False (1>0)

---

Speed execution compared to groupby and defaultdict solutions:

import random
from collections import defauldict
from itertools import groupby

list_balls = []
for _ in range(10000000):
    list_balls.append(
        {
            'source': random.choice(['store', 'donation']),
            'id': random.randint(0,10000),
            'air': random.randint(0,4)
        }
    )

def vanilla_filter_list(list_balls):
    result = {}

    for e in list_balls:
        if e['id'] not in result or (
              (e['air'], e['source'] == 'store') > 
              (result[e['id']]['air'], result[e['id']]['source'] =='store')
            ):
            result[e['id']] = e

    return list(result.values())

def groupby_filter_list(list_balls):
    return [max(list(v), 
                key=lambda x: (x["air"], x["source"] == "store")) for k, v in groupby(
        sorted(list_balls, key=lambda x: x["id"]),
        lambda x: x["id"])]

def collections_filter_list(list_balls):
    d = defaultdict(list)
    for ball in list_balls:
        d[ball["id"]].append(ball)

    return [
        max(group, key=lambda x: (x["air"], x["source"] == "store")) for group in d.values()
    ]

%%time
vanilla_filter_list(list_balls) # 5.52s

%%time
groupby_filter_list(list_balls) #14.3s

%%time
collections_filter_list(list_balls) #8.41s

Answer 3

Try this :

all_id = set(i['id'] for i in list_balls)
new_list_ballls = []
for id_ in all_id:
    max_air = max(i['air'] for i in list_balls if i['id']==id_)
    max_air_count = sum(1 for i in list_balls if i['air']==max_air and i['id']==id_)
    if max_air_count==1:
        for i in list_balls:
            if i['id']==id_ and i['air']==max_air:
                new_list_ballls.append(i)
    else:
        for i in list_balls:
            if i['id']==id_ and i['air']==max_air and i['source'] != 'store':
                new_list_ballls.append(i)

Output :

[{'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75}, 
{'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}]

Answer 4

Here

from collections import defaultdict

list_balls = [{'id': '803371', 'is_used': False, 'source': 'store', 'air': 0.9},
              {'id': '803371', 'is_used': False, 'source': 'donation', 'air': 0.20},
              {'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75},
              {'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}]

grouped_data = defaultdict(list)

for entry in list_balls:
    grouped_data[entry['id']].append(entry)

final_list = []

for k, v in grouped_data.items():
    if len(v) == 1:
        final_list.append(v[0])
    else:
        # sort by air
        x = sorted(v, key=lambda k1: k1['air'], reverse=True)
        if x[0]['air'] != x[1]['air']:
            final_list.append(x[0])
        else:
            # decide by source
            if [x[0]]['source'] == 'store':
                final_list.append(x[0])
            elif [x[1]]['source'] == 'store':
                final_list.append(x[1])

for entry in final_list:
    print(entry)

output

{'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}
{'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75}

Answer 5

I would first group by id with a defaultdict then get the maximum dictionary by air afterwards. If a tie occurs with air and id , then use source as a secondary key for max() .

Demo:

from collections import defaultdict

list_balls = [
    {"id": "803371", "is_used": False, "source": "store", "air": 0.9},
    {"id": "803371", "is_used": False, "source": "donation", "air": 0.20},
    {"id": "30042", "is_used": False, "source": "donation", "air": 0.75},
    {"id": "803371", "is_used": False, "source": "store", "air": 1},
    {"id": "803371", "is_used": False, "source": "donation", "air": 1},
]

d = defaultdict(list)
for ball in list_balls:
    d[ball["id"]].append(ball)

result = [
    max(group, key=lambda x: (x["air"], x["source"] == "store")) for group in d.values()
]

print(result)

Output:

[{'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}, {'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75}]

Answer 6

Nothing needless, only a pure python, almost.
Sort the list of dictionaries by id , then by negative values of air so that the largest ones go first, and then by source so that the entries with store go first. After that, the first entry is selected from each set of dictionaries, which are grouped by id .

import pprint

list_balls = [
  {'id': '803371', 'is_used': False, 'source': 'store', 'air': 0.9},
  {'id': '803371', 'is_used': False, 'source': 'donation', 'air': 0.20},
  {'id': '30042', 'is_used': False, 'source': 'donation', 'air': 0.75},
  {'id': '803371', 'is_used': False, 'source': 'store', 'air': 1}
]
list_balls.sort(key=lambda k: (k['id'], -k['air'], 0 if k['source'] == 'store' else 1))
pprint.pprint([d for i, d in enumerate(list_balls) if i == 0 or list_balls[i - 1]['id'] != d['id']])

Output:

[{'air': 0.75, 'id': '30042', 'is_used': False, 'source': 'donation'},
 {'air': 1, 'id': '803371', 'is_used': False, 'source': 'store'}]