根据2个键在词典列表中删除重复的词典

Question

I have a list of dictionaries, like this:

   my_list = [{'key1':'1', 'date':'2015-01-09'}, {'key1':'3', 'date':'2015-01-09'}, {'key1':'1', 'date':'2014-03-19'}, \
       {'key1':'4', 'date':'2015-05-09'} ,...]

In some of the dictionaries, the value of key1 are repeated and I want to remove them from the list based on date (another key of the dictionary) and keep only the dictionaries that have the earliest date. Result:

    my_list = [{'key1':'3', 'date':'2015-01-09'}, {'key1':'1', 'date':'2014-03-19'}, {'key1':'4', 'date':'2015-05-09'} ,...]

Performance is important.

Answer 1

I would rebuild a dictionary with key1 as key in a dictionary comprehension, using sorted values (reversed) so earliest date are returned last, overwriting same keys: only the earliest date remains:

my_list = [{'key1':'1', 'date':'2015-01-09'}, {'key1':'3', 'date':'2015-01-09'}, {'key1':'1', 'date':'2014-03-19'}, \
       {'key1':'4', 'date':'2015-05-09'}]

my_dict = {d["key1"]:d for d in sorted(my_list,key=lambda l:l["date"],reverse=True)}

print(list(my_dict.values()))

result (I supposed that ordering didn't matter, or else I cannot use the dictionary, since order is not preserved):

[{'key1': '1', 'date': '2014-03-19'}, {'key1': '3', 'date': '2015-01-09'}, {'key1': '4', 'date': '2015-05-09'}]

(note that sorting the dates with lexicographical order is OK because they're YYYY-MM-DD format and it makes things easier: no need to parse the dates)

An alternate solution if you're short in memory would be to avoid the sorting part because it creates a sorted copy of the list beforehand (doesn't duplicate the data, but still, it can eat some memory).

In that case, a classical loop will do, slower but less memory-hungry (and no sorting needed). Using get with a default value to return 'A' when the key isn't in the destination dictionary to force insertion ( A ranks higher than any digit).

my_dict = {}

for l in my_list:
    k = l['key1']
    d = l['date']

    if my_dict.get(k,'A') > d:
        my_dict[k] = d

Answer 2

Both of the answers work, I think though when I was a real beginner I would have preferred something a little simpler. What I would do is similar to @Jean_Francois's answer but I think is a bit simpler ( though it has more lines of code)

I would build a dictionary from the list and as I add to it I would check the date. The data checking is easy as he noted

from collections import defaultdict
min_date_dict = defaultdict(dict)
for item_date in my_list:
    key = item_date['key1']
    date = item_date['date']
    if key in min_date:
          if min_date[key]['date'] > date:
                min_date[key] = item_date
    else:
       min_date[key] = item_date

This transformation places your items into a dictionary with the key as the value of key1

defaultdict(<type 'dict'>, {'1': {'date': '2014-03-19', 'key1': '1'}, '3': {'date': '2015-01-09', 'key1': '3'}, '4': {'date': '2015-05-09', 'key1': '4'}})

now to put it back into a list

item_date_list = min_date.values()

Answer 3

import pandas as pd

list(pd.DataFrame(my_list).sort_values(by='date').drop_duplicates(subset=['key1'], keep='first').apply(lambda s: s.to_dict(), axis=1).values)

Answer 4

Here is a more verbose way to do it

my_list = [{'key1':'1', 'date':'2015-01-09'}, 
           {'key1':'3', 'date':'2015-01-09'}, 
           {'key1':'1', 'date':'2014-03-19'},
           {'key1':'4', 'date':'2015-05-09'}]

mins = {}
for i, d in enumerate(my_list):
    if d['key1'] not in mins or mins[d['key1']]['date'] > d['date']:
            mins[d['key1']] = {'date': d['date'], 'ind': i}

indices = sorted([d['ind'] for d in mins.values()])
filtered = [my_list[i] for i in indices]
print(filtered)

Answer 5

your can use itertools groupby to group by keys and then takes the minimum date for each group. see example below

final_list = [min(list(g), key = lambda x: x['date']) for k, g in groupby(sorted(my_list, key=lambda x: x['key1']), lambda x: x['key1'])]

results in

[{'date': '2014-03-19', 'key1': '1'}, {'date': '2015-01-09', 'key1': '3'}, {'date': '2015-05-09', 'key1': '4'}]