For a list of lists, merge the dictionaries with collections.defaultdict

Question

This is a simple question, but I'm confused by the behavior of collections.defaultdict . This is to help me learn how this works.

This question is an extrapolation from this useful question: How to merge a list of multiple dictionaries into a dictionary of lists?

Let's now say I have a list of lists of dictionaries. I want to merge the dictionaries as detailed in the above question:

list_of_dictionaries2 = [[{0:3523, 1:3524, 2:3540, 4:3541, 5:3542}, 
    {0:7245, 1:7246, 2:7247, 3:7248, 5:7249, 6:7250},
    {1:20898, 2:20899, 3:20900, 4:20901, 5:20902}], [{0:3, 1:4, 2:5, 3:6}]]

The intended answer is this:

correct2 = [[{0:[3523, 7245], 1:[3524, 7246, 20898], 2:[3540, 7247, 20899], 
            3:[7248, 20900], 4:[3541, 20901], 5:[3542, 7249, 20902], 6:[7250]}], 
            [{0:3, 1:4, 2:5, 3:6}]]

Previously, for a single list of dictionaries, we solved this by creating an empty dictionary with default values as lists, ie we used collections.defaultdict(list) .

Given this case is a list of lists, I thought another for loop would be the solution, appending dictionaries into an empty list:

from collections import defaultdict
correct2 = defaultdict(list)

empty = []

for smaller_list in list_of_dictionaries2:
    for d in smaller_list:
        for k,v in d.items():
            correct2[k].append(v)
    empty.append(correct2)

This is very wrong.

>>> print(empty)
[defaultdict(<class 'list'>, {0: [3523, 7245, 3], 1: [3524, 7246, 20898, 4], 
2: [3540, 7247, 20899, 5], 4: [3541, 20901], 5: [3542, 7249, 20902], 
3: [7248, 20900, 6], 6: [7250]}), defaultdict(<class 'list'>, 
{0: [3523, 7245, 3], 1: [3524, 7246, 20898, 4], 2: [3540, 7247, 20899, 5], 
4: [3541, 20901], 5: [3542, 7249, 20902], 3: [7248, 20900, 6], 6: [7250]})]

It looks like all dictionaries were combined. And there are two copies. This is not what I want.

How do I do this for each individual list, like above? Where am I mistaken in my understanding?

Answer 1

You don't actually have a list of dictionaries, but a list of lists of dictionaries, and you're trying to merge the dictionaries within the sub-lists, so you should initialize your defaultdict inside the loop that iterates through the main list:

empty = []
for smaller_list in list_of_dictionaries2:
    correct2 = defaultdict(list)
    for d in smaller_list:
        for k,v in d.items():
            correct2[k].append(v)
    empty.append(correct2)

empty would become:

[defaultdict(<class 'list'>, {0: [3523, 7245], 1: [3524, 7246, 20898], 2: [3540, 7247, 20899], 4: [3541, 20901], 5: [3542, 7249, 20902], 3: [7248, 20900], 6: [7250]}), defaultdict(<class 'list'>, {0: [3], 1: [4], 2: [5], 3: [6]})]

Note that your expected output for the second merged defaultdict is incorrect since the value of each key should be a list after merge. Also the list of lists of dictionary should become a list of defaultdict s of lists after the merge, not a list of lists of defaultdict s of lists.

Answer 2

you should initialize correct2 every loop like this

for smaller_list in list_of_dictionaries2:
    correct2 = defaultdict(list)
    for d in smaller_list:
        for k, v in d.items():
            correct2[k].append(v)
    empty.append(correct2)

out put will be

[defaultdict(<class 'list'>, {0: [3523, 7245], 1: [3524, 7246, 20898],
2: [3540, 7247, 20899], 4: [3541, 20901], 5: [3542, 7249, 20902], 
3: [7248, 20900], 6: [7250]}), 
defaultdict(<class 'list'>, {0: [3],1: [4], 2: [5], 3: [6]})]

note that defaultdict is mutable Objects