How to remove duplicates from list of dictionaries in python

Question

I have 2 list of dictionaries with the same values. I want to remove 1 value.

This is how it looks:

[
{
  "activity_name": "Wait Time",
  "task_detail": [
    {
      "created_date": "2021-04-29 10:00:25.695254",
      "row_id": ""
    }
  ]
},
{
  "activity_name": "Wait Time",
  "task_detail": [
    {
      "created_date": "2021-04-29 10:28:42.131017",
      "row_id": ""
    }
  ]
}
]

Here is my code:

result=[]
for gc_item in gc_response['Items']:
    for sch_act in gc_item['scheduled_activity']:
        result.append(sch_act)

How to remove duplicate value for this? Under Scheduled_activity only i have the value(Activity name and task_detail)

I need to remove 1 of the duplicate value. The output should be:

{
  "activity_name": "Wait Time",
  "task_detail": [
    {
      "created_date": "2021-04-29 10:28:42.131017",
      "row_id": ""
    }
  ]
}

Answer 1

You can only add if the activity_name doesn't already exist in list

result=[]
acitivity_names_added = []
for sch_act in gc_item['scheduled_activity']:
    if(sch_act['activity_name'] not in acitivity_names_added):
        result.append(sch_act) 
        acitivity_names_added.append(sch_act['activity_name'])

If you want to keep the last one instead of the first one then iterate through this loop in reverse order

result=[]
acitivity_names_added = []
for sch_act in gc_item['scheduled_activity'][::-1]:
    if(sch_act['activity_name'] not in acitivity_names_added):
        result.append(sch_act) 
        acitivity_names_added.append(sch_act['activity_name'])

Answer 2

This is similar to Remove duplicate dict in list in Python

In your case, task_detail seems to be more complicated, so the easy tuple-method like [dict(t) for t in {tuple(d.items()) for d in l}] may not work, but you can try the json method in the same thread:

import json

result = []
for gc_item in gc_response['Items']:
    set_of_jsons = {json.dumps(d, sort_keys=True) for d in gc_item['scheduled_activity']}
    result += [json.loads(t) for t in set_of_jsons]

It basically turns list/dict to string so that you could remove duplicates by set.