How to merge items belonging to the same key in list of dictionaries in python?

Question

Here is an example of the json file

[
    {
        "id": "id",
        "key1": "value1",
        "key2": [
            {
                "next_id": "next_id1"
            },
            {
                "next_id": "next_id2"
            }
        ]
    },
    {
        "id": "id",
        "key1": "value1",
        "key2": [
            {
                "next_id": "next_id3"
            },
            {
                "next_id": "next_id4"
            }
        ]
    }
]

And I want the output to look like

[
    {
        "id": "id",
        "key1": "value1",
        "key2": [
            {
                "next_id": "next_id1"
            },
            {
                "next_id": "next_id2"
            },
            {
                "next_id": "next_id3"
            },
            {
                "next_id": "next_id4"
            }
        ]
    }
]

I am merging key2 based on the same values of id . How do I achieve in python?

I believe in java you can do this via fullQuery but trying to figure out the most efficient way to do in python.

Answer 1

You could use pandas:

Input:

l = [
    {
        "id": "id",
        "key1": "value1",
        "key2": [
            {
                "next_id": "next_id1"
            },
            {
                "next_id": "next_id2"
            }
        ]
    },
    {
        "id": "id",
        "key1": "value1",
        "key2": [
            {
                "next_id": "next_id3"
            },
            {
                "next_id": "next_id4"
            }
        ]
    }
]

Code:

import pandas as pd
df = pd.DataFrame(l)
df = df.groupby(['id', 'key1'])['key2'].apply(list).reset_index()
df['key2'] = df['key2'].map(lambda z: [y for x in z for y in x])
df.to_dict(orient='row')

Output:

[{'id': 'id',
  'key1': 'value1',
  'key2': [{'next_id': 'next_id1'},
   {'next_id': 'next_id2'},
   {'next_id': 'next_id3'},
   {'next_id': 'next_id4'}]}]

Answer 2

Not the most elegant solution but I think this should handle all your use cases including possible nested dicts.

def mergeDictRecurse(D1, D2):
  if not D1 or not D2:
    return D1 if not D2 else D2
  D3 = D1.copy()
  for key, value in D1.items():
    if key in D2:
      if type(value) is dict:
        mergeDictRecurse(D1[key], D2[key])
      else:
        if type(value) in (int, float, str):
          D3[key] = [value]
        if type(D2[key]) is list:
          # If list of items, make sure the item doesn't
          # already exist in original dict.
          for item in D2[key]:
            if item not in D2[key]:
              D3[key].extend(item)
            else:
              continue
        else:
          D3[key].append(D2[key])
          D3[key] = list(set(D3[key]))  # Get rid of duplicate elements in a list.
          D3[key] = D3[key][0] if len(D3[key]) == 1 else D3[key]  # Convert back to element if only one item.
  for key, value in D2.items():
    if key not in D1:
      D3[key] = value
  return D3

def mergeDictsInList(list_dicts):
  merged_dict = {}
  for idx, unmerged_dict in enumerate(list_dicts):
      merged_dict = mergeDictRecurse(merged_dict, list_dicts[idx])
  return [merged_dict]

Use:

merged_dict = mergeDictsInList(list_json)

list_json:

[{'id': 'id',
  'key1': 'value1',
  'key2': [{'next_id': 'next_id1'}, {'next_id': 'next_id2'}]},
 {'id': 'id',
  'key1': 'value1',
  'key2': [{'next_id': 'next_id3'}, {'next_id': 'next_id4'}]}]

merged_dict:

[{'id': 'id', 'key1': 'value1', 'key2': [{'next_id': 'next_id1'}, {'next_id': 'next_id2'}, {'next_id': 'next_id3'}, {'next_id': 'next_id4'}]}]