Python list of dict into a 2D list

Question

Input

Couldn't get my head round this: I've got a python list of dictionaries like so:

d = [{'a': ['a1', 'a2', 'a3'], 'b': ['b1', 'b2']}, {...}, {...}]

Important notes:

• number of keys ('a', 'b'...) is const (eg from A to Z)
• lengths of values (lists) is arbitrary, it may even be an empty list
• elements in values don't matter and are here shown as 'a1', 'b2' for clarity only

Expected result

I need to flatten the nested lists into a 2D list of separate dicts like so:

d = [ 
        [ {'key': 'a', 'val': 'a1'}, {'key': 'b', 'val': 'b1'}, {...} ],
        [ {'key': 'a', 'val': 'a2'}, {'key': 'b', 'val': 'b2'}, {...} ],
        [ {'key': 'a', 'val': 'a3'}, {'key': 'b', 'val': 'b3'}, {...} ],
        [ ... ]
    ]

That is, each dict is atomic containing to items: 'key' that stands for the original keys, and 'val' - the original lists broken element wise. Since the number of keys is const, the inner lists must have equal lengths. So this is basically a permutation problem.

Note Since the number is elements in the values (lists) is not const, the permutations must be able to pad the shorter lists with None or use the end elements. For example:

[
  [ {'key': 'a', 'val': 'a5'}, # -- last possible value for 'a' =5
    {'key': 'b', 'val': 'b100'}, # -- but 'b' has at least 100 values...
    ...
  ],
  [ {'key': 'a', 'val': 'a5'}, # -- ...so the next list uses the same val for 'a'
    {'key': 'b', 'val': 'b101'}, # -- ...and a next one for 'b'
    ...
  ],
]

Tried pandas.to_dict and various native python tricks, but just couldn't succeed... For example, pandas.to_dict('list') gives

{ 'keys': ['a', 'b', ...], 'vals': [[a1, a2, ...], ['b1', 'b2', ...]]}

Answer 1

OK, got it finally. Here's the solution -- use Cartesian product with itertools.product :

import itertools, json

def flatten(d, sortkeys=True):
    keys = sorted(d.keys()) if sortkeys else d.keys()
    return [
        {k: v for k, v in zip(keys, list_prod_value)}
        for list_prod_value in itertools.product(*(d[k] for k in keys if d[k]))
    ]

d = {1: [1, 2, 3, 4], 2: [5, 6], 3: [7]}

print(json.dumps(flatten(d), indent=2))

Output:

[
  {
    "1": 1,
    "2": 5,
    "3": 7
  },
  {
    "1": 1,
    "2": 6,
    "3": 7
  },
  {
    "1": 2,
    "2": 5,
    "3": 7
  },
  {
    "1": 2,
    "2": 6,
    "3": 7
  },
  {
    "1": 3,
    "2": 5,
    "3": 7
  },
  {
    "1": 3,
    "2": 6,
    "3": 7
  },
  {
    "1": 4,
    "2": 5,
    "3": 7
  },
  {
    "1": 4,
    "2": 6,
    "3": 7
  }
]