Unique values of list of json objects

Question

I have a huge list of json objects, and many of them are repeated. Those that are repeated have exactly the same values for the same keys. For example in the following list

[{ "name": "John", "id": 1},{ "name": "Carl", "id": 5},{ "name": "John", "id": 1}]

I want to get the unique objects

[{ "name": "John", "id": 1},{ "name": "Carl", "id": 5}]

I tried using the function set() but apparently it does not work with elements of type: dict.

Is there any efficient way of doing this?

Thanks in advance

Answer 1

Try hashing the dictionaries before adding them to your set. To do this, simply turn the dictionary into a string.

d1 = { "name": "John", "id": 1}
d2 = { "name": "Carl", "id": 5}
d3 = { "name": "John", "id": 1}
s = set()

for d in [d1,d2,d3]:
    if str(d) not in s:
        s.add(str(d))