How to remove duplicates from the dictionary if values repeats

Question

I have the following list of dict:

test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn@gmail.com', 'Account': 'Product' }, { 'id': '219', 'Name': 'umar', 'Email': 'ddhi@gmail.com', 'Account': 'Product' }, { 'id': '74', 'Name': 'Are', 'Email': 'ddhit@gmail.com', 'role': 'Tester' }, { 'id': '24', 'Name': 'Mee', 'Email': 'huul@gmail.com', 'role': 'Tester' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' }, { 'id': '220', 'Name': 'Sdy', 'Email': 'deyan@gmail.com', 'role': 'Product' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' } ] 

I want to remove the dictionary if the email repeats

Code is below

[dict(element) for element in {tuple(each.items()) for each in test }]

My code is working only all the keys and values are same. My requirement is delete the duplicates dictionary for the email

Answer 1

seen = set()
result = [d for d in test if d['Email'] not in seen and not seen.add(d['Email'])]

Answer 2

You could also use pandas in the following way:

import pandas as pd
test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn@gmail.com', 'Account': 'Product' }, { 'id': '219', 'Name': 'umar', 'Email': 'ddhi@gmail.com', 'Account': 'Product' }, { 'id': '74', 'Name': 'Are', 'Email': 'ddhit@gmail.com', 'role': 'Tester' }, { 'id': '24', 'Name': 'Mee', 'Email': 'huul@gmail.com', 'role': 'Tester' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' }, { 'id': '220', 'Name': 'Sdy', 'Email': 'deyan@gmail.com', 'role': 'Product' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' } ] 

df = pd.DataFrame(test)
df_res = df.drop_duplicates('Email')

From your question, it isn't clear how you want to choose the entries to keep (first instance or something else) but you could apply some more sophisticated method of dropping using pandas.

Answer 3

You need to create a new list with only the unique items as below:

unique_emails=[]

test_with_unique_emails = []

for item in test:
    if item["Email"] not in unique_emails:
        test_with_unique_emails.append(item)
        unique_emails.append(item["Email"])

Answer 4

dict([(d['Email'], d) for d in test]).values()

Answer 5

You can use something like that

test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn@gmail.com', 'Account': 'Product' }, { 'id': '219', 'Name': 'umar', 'Email': 'ddhi@gmail.com', 'Account': 'Product' }, { 'id': '74', 'Name': 'Are', 'Email': 'ddhit@gmail.com', 'role': 'Tester' }, { 'id': '24', 'Name': 'Mee', 'Email': 'huul@gmail.com', 'role': 'Tester' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' }, { 'id': '220', 'Name': 'Sdy', 'Email': 'deyan@gmail.com', 'role': 'Product' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' } ] <br>
unique_emails = []
final = []

for element in test:
    if element['Email'] not in unique_emails:
        final.append(element)

print(final)

Also

import ast

test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn@gmail.com', 'Account': 'Product' }, { 'id': '219', 'Name': 'umar', 'Email': 'ddhi@gmail.com', 'Account': 'Product' }, { 'id': '74', 'Name': 'Are', 'Email': 'ddhit@gmail.com', 'role': 'Tester' }, { 'id': '24', 'Name': 'Mee', 'Email': 'huul@gmail.com', 'role': 'Tester' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' }, { 'id': '220', 'Name': 'Sdy', 'Email': 'deyan@gmail.com', 'role': 'Product' }, { 'id': '230', 'Name': 'San', 'Email': 'deyan@gmail.com', 'role': 'Tester' } ]

print([ast.literal_eval(el1) for el1 in set([str(el2) for el2 in test])])

Answer 6

emails = []
for t in test:
    if t['Email'] in emails:
            emails.append(t['Email'])
    else:
            test.remove(t)

test will now have the latest list of unique dictionaries.