根据键的值过滤字典列表

Question

if two values are identical in a list of dictionaries, I would like the list filtered with only one of the dictionaries. I do not care about the second (or third dictionary that matches).

crcs = [
        {'compress_name': 'file1.bin', 'crc': '55A0669C', 'name': 'R:\\filepath\\system\\compress1.zip'},
        {'compress_name': 'file3.bin', 'crc': '55A0669C', 'name': 'R:\\filepath\\system\\compress2.zip'},
        {'compress_name': 'file2.bin', 'crc': '66B07710', 'name': 'R:\\filepath\\system\\compress2.zip'},
        {'compress_name': 'file5.bin', 'crc': '66B07710', 'name': 'R:\\filepath\\system\\compress3.zip'}
    ]

expected results is a list of two dictionaries with differing "crc" values.

[
        {'compress_name': 'file1.bin', 'crc': '55A0669C', 'name': 'R:\\filepath\\system\\compress1.zip'},
        {'compress_name': 'file2.bin', 'crc': '66B07710', 'name': 'R:\\filepath\\system\\compress2.zip'},
    ]

or any other combination of the CRC values matching 55A0669C and 66B07710. The list of dictionaries could be 400 or more items long.

I'm using python 3.7

Answer 1

if it's only crc what need to be unique, then you can use

crcs = [ {'compress_name': 'file1.bin', 'crc': '55A0669C', 'name': 'R:\filepath\system\compress1.zip'}, {'compress_name': 'file3.bin', 'crc': '55A0669C', 'name': 'R:\filepath\system\compress2.zip'}, {'compress_name': 'file2.bin', 'crc': '66B07710', 'name': 'R:\filepath\system\compress2.zip'}, {'compress_name': 'file5.bin', 'crc': '66B07710', 'name': 'R:\filepath\system\compress3.zip'} ]

crcs_all = []
crcs_uniq = []

for i in range(len(crcs)):
    crc = crcs[i]['crc']
    if crc not in crcs_all:
        crcs_all.append(crc)
        crcs_uniq.append(crcs[i])

print(crcs_uniq)

That will give you

    [ {'compress_name': 'file1.bin', 'crc': '55A0669C', 'name': 'R:\x0cilepath\\system\\compress1.zip'}, 
      {'compress_name': 'file2.bin', 'crc': '66B07710', 'name': 'R:\x0cilepath\\system\\compress2.zip'}]

Answer 2

Solution

You could use caste the list of dictionaries into a dataframe and then select the unique crc values. Finally, you could get the first occurences of the duplicate crc values by using list.index(crc) and store than in a list unique_idx . We use this unique_idx to filter out the relevant rows from the dataframe df and then extract that data as a dict .

Short Solution

import pandas as pd

df = pd.DataFrame(crcs)
unique_crcs = df.crc.unique().tolist()
unique_idx = []
for crc in unique_crcs:
    unique_idx.append(all_crcs.index(crc))

dfu = df.iloc[unique_idx]
dfu.T.to_dict()

Output :

{0: {'compress_name': 'file1.bin',
  'crc': '55A0669C',
  'name': 'R:\\filepath\\system\\compress1.zip'},
 2: {'compress_name': 'file2.bin',
  'crc': '66B07710',
  'name': 'R:\\filepath\\system\\compress2.zip'}}

Detailed Solution

1. Make Data

import pandas as pd

crcs = [{'compress_name': 'file1.bin', 'crc': '55A0669C', 'name': r'R:\filepath\system\compress1.zip'}, 
        {'compress_name': 'file3.bin', 'crc': '55A0669C', 'name': r'R:\filepath\system\compress2.zip'}, 
        {'compress_name': 'file2.bin', 'crc': '66B07710', 'name': r'R:\filepath\system\compress2.zip'}, 
        {'compress_name': 'file5.bin', 'crc': '66B07710', 'name': r'R:\filepath\system\compress3.zip'} ]

df = pd.DataFrame(crcs)
print(df)

Output :

  compress_name       crc                              name
0     file1.bin  55A0669C  R:\filepath\system\compress1.zip
1     file3.bin  55A0669C  R:\filepath\system\compress2.zip
2     file2.bin  66B07710  R:\filepath\system\compress2.zip
3     file5.bin  66B07710  R:\filepath\system\compress3.zip

Select Unique CRC Rows

unique_crcs = df.crc.unique().tolist()
all_crcs = df.crc.to_list()

unique_idx = []
uniques = dict()
for crc in unique_crcs:
    idx = all_crcs.index(crc)
    uniques.update({crc: idx})
    unique_idx.append(idx)

print(uniques)
print(all_crcs)

Output :

{'55A0669C': 0, '66B07710': 2}
['55A0669C', '55A0669C', '66B07710', '66B07710']

Make Dict with Unique CRC Redords Only

dfu = df.iloc[unique_idx]
dfu.T.to_dict()

Output :

{0: {'compress_name': 'file1.bin',
  'crc': '55A0669C',
  'name': 'R:\\filepath\\system\\compress1.zip'},
 2: {'compress_name': 'file2.bin',
  'crc': '66B07710',
  'name': 'R:\\filepath\\system\\compress2.zip'}}