Python to iterate list of dictionary and merge keys whilst retaining values

Question

I'm trying to iterate over a list of dictionaries and merge on a duplicate key (in this case called 'id'), moving the value (of the duplicate key 'url'), from that duplicate instance, to the first (retained) instance as a list of values.

So, to go from this:

[{'id': ‘XYZ', 'url': ‘THIS IS A URL'}, 
{'id': ‘XYZ', 'url': ‘THIS IS A URL'},  
{'id': ‘XYZ', 'url': ‘THIS IS A URL'},  
{'id': ‘ABC', 'url': ‘THIS IS A URL'},  
{'id': ‘DEF', 'url': ‘THIS IS A URL'}]

To this:

[{'id': ‘XYZ', 'url': ‘THIS IS A URL', 'THIS IS A URL', 'THIS IS A URL'},
{'id': ‘ABC', 'url': ‘THIS IS A URL'},
{'id': ‘DEF', 'url': ‘THIS IS A URL'}]

Answer 1

You may use a defaultdict to map between ids to a list of the urls, then convert it back to your original object.

import collections
id_to_url = collections.defaultdict(list)

urls = [{'id': 'XYZ', 'url': 'THIS IS A URL'}, 
{'id': 'XYZ', 'url': 'THIS IS A URL'},  
{'id': 'XYZ', 'url': 'THIS IS A URL'},  
{'id': 'ABC', 'url': 'THIS IS A URL'},  
{'id': 'DEF', 'url': 'THIS IS A URL'}]

for url in urls:
    id_to_url[url['id']].append(url['url'])

urls = [{'id': key, 'url': value} for key, value in id_to_url.items()]

result:

>>> urls
[{'id': 'XYZ', 'url': ['THIS IS A URL', 'THIS IS A URL', 'THIS IS A URL']}, {'id': 'ABC', 'url': ['THIS IS A URL']}, {'id': 'DEF', 'url': ['THIS IS A URL']}]