Remove duplicates values from list of dictionaries in python

Question

I want to remove duplicates values from list which is inside the dictionary. I am trying to make configurable code to work on any field instead of making field specific.

Input Data:

{'Customer_Number': 90617174, 'Email': [{'Email_Type': 'Primary', 'Email': ['saman.zonouz@rutgers.edu', 'saman.zonouz@rutgers.edu']}], 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280, 12177218280]}]}

Expected Output Data:

{'Customer_Number': 90617174, 'Email': [{'Email_Type': 'Primary', 'Email': ['saman.zonouz@rutgers.edu']}], 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280]}]}

code tried:

dic = {'Customer_Number': 90617174, 'Email': [{'Email_Type': 'Primary', 'Email': ['saman.zonouz@rutgers.edu', 'saman.zonouz@rutgers.edu']}], 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280, 12177218280]}]}

res = []
for i in dic:
    if i not in res:
        res.append(i)
  

Answer 1

You can use set()

import json 

dic = {
    'Customer_Number': 90617174,
    'Email': [
        {
            'Email_Type': 'Primary',
            'Email': list(set([
                'saman.zonouz@rutgers.edu',
                'saman.zonouz@rutgers.edu',
            ]))
        }
    ],
    'Phone_Number': [
        {
            'Phone_Type': 'Mobile',
            'Phone': list(set([
                12177218280,
                12177218280,
            ]))
        }
    ]
}

print(json.dumps(dic,indent=2))

If you want to do it on a list of dic 's then you can do like this:

for dic in dics:
    for email in dic['Email']:
        email['Email'] = list(set(email['Email']))
    
    for phone in dic['Phone_Number']:
        phone['Phone'] = list(set(phone['Phone']))

Answer 2

The approach that you started with, you need to go a few levels deeper with that to find every such "repeating" list and dedupe it.

To dedupe, you can use a set - which is also a "container" data structure like a list but with some (many?) differences. You can get a good introduction to all of this in the official python docs -

for key in dic:
    if isinstance(dic[key], list):
        for inner_dict in dic[key]:
            for inner_key in inner_dict:
                if isinstance(inner_dict[inner_key], list):
                    inner_dict[inner_key] = list(set(inner_dict[inner_key]))

print(dic)
#{'Customer_Number': 90617174,
# 'Email': [{'Email_Type': 'Primary', 'Email': ['saman.zonouz@rutgers.edu']}],
# 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280]}]}