List Comprehension to Flatten a Dictionary of Dictionaries

Question

I have a dictionary of dictionaries:

my_dict = {
    'a': {(1,2): True,
          (1,3): False},
    'b': {(1,4): True,
          (2,3): False}
}

The dictionary is always of this form, but every 'child' dictionary has a different set of keys: my_dict['a'][(1,2)] exists, but that doesn't mean my_dict['b'][(1,2)] also exists.

I want a list (in no particular order) of the boolean values:

[True, False, True, False]

I am trying to use a single list comprehension to accomplish this:

[my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]

This raises an error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-142-dc1565efcdc8> in <module>()
      6 }
      7 
----> 8 [my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]
KeyError: (2, 3)

It appears to be looking for (2,3) in both my_dict['a'] and my_dict['b']. I thought the comprehension I wrote would only look for the keys in the appropriate dictionary.

I've seen this solution which can work to flatten any nested dictionary. I also know I could brute force it with imperative loops. I am just trying to understand why the list comprehension isn't working the way I have it written.

Answer 1

You want to loop over the values of the values:

[v for nested in outer.itervalues() for v in nested.itervalues()]

Note that the loops need to be ordered the way you'd nest them; outer loop first:

for nested in outer.itervalues():
    for v in nested.itervalues():
        # use v

You had the order mixed up; your code only gave KeyError because you had a pre-existing letter global.

Demo:

>>> my_dict = {
...     'a': {(1,2): True,
...           (1,3): False},
...     'b': {(1,4): True,
...           (2,3): False}
... }
>>> [v for nested in my_dict.itervalues() for v in nested.itervalues()]
[True, False, False, True]

Answer 2

As elsherbini said,

[my_dict[letter][pair] for letter in my_dict for pair in my_dict[letter]]

This also works:

[little_dict[k] for little_dict in [my_dict[letter] for letter in my_dict] for k in little_dict]

Both produce [True, False, False, True] .

You want to understand why your original try doesn't work.

[my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]

The only reason this runs at all is that you must have had letter previously defined, perhaps a definition left over from previously running some similar comprehension. It begins by trying to interpret for pair in my_dict[letter] and cannot make any sense of this unless letter was already defined. If letter was previously defined as b (value left over from running a previous list comprehension) then it sets pair to the keys of my_dict['b'] . It then looks at for letter in my_dict and sets letter to 'a' and to 'b' . It then tries to evaluate the first part, my_dict[letter][pair] , but it's using the keys from b so this won't work when letter takes the value 'a' .

Below, I run your comprehension and get NameError , then run another comprehension which as a side-effect sets the value of letter , then I run your same comprehension again and get KeyError .

Python 2.6.9 
>>> my_dict = {
...     'a': {(1,2): True,
...           (1,3): False},
...     'b': {(1,4): True,
...           (2,3): False}
... }
>>> 
>>> [my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'letter' is not defined
>>> letter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'letter' is not defined
>>> [letter for letter in my_dict]
['a', 'b']
>>> letter
'b'
>>> [my_dict[letter][pair] for pair in my_dict[letter] for letter in my_dict]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: (2, 3)
>>> 

Note in the above that the KeyError only happens after having (accidentally) set the value of letter . The first run produces NameError instead.