Filtering List of Dict based of a list of keys in python

Question

I am trying to filter a list of dict based on another list of keys. See the below sample.

# Input list of dict
lst_dict=[{'TABLE_NAME':'TEST1','COLUMN_NAME':'TEST_COLUMN','COLUMN_DATA_TYPE': 'varchar'},
        {'TABLE_NAME': 'TEST1', 'COLUMN_NAME': 'TEST_COLUMN2', 'COLUMN_DATA_TYPE': 'varchar'}, 
        {'COLUMN_DATA_TYPE': 'varchar','TABLE_NAME': 'TEST2','COLUMN_NAME': 'TEST_COLUMN1'}]
# Key List
key_lst = ['TABLE_NAME','COLUMN_NAME']

My expected output:

[{'TABLE_NAME': 'TEST2', 'COLUMN_NAME': 'TEST_COLUMN1'}, {'TABLE_NAME': 'TEST1', 'COLUMN_NAME': 'TEST_COLUMN'}, {'TABLE_NAME': 'TEST1', 'COLUMN_NAME': 'TEST_COLUMN2'}]

I wrote the below code but it is only returning only 1 dict in the list instead of 3. (as per my example)

new_list = [{k:v for i in lst_dict for k,v in i.items() if k in key_lst}]

I am not sure where I am wrong.. Any suggestion is appreciated!

Answer 1

The way you have written new_list , it looks like you expect it to be a list comprehension, but it is only a list containing a single item, which is a dict comprehension. You probably want to move the iterator over lst_dict out of the dictionary and into the list. It should look like this:

new_list = [{k:v for k,v in i.items() if k in key_lst} for i in lst_dict]

Answer 2

You may use list comprehension with dict comprehension as:

>>> my_list = [{key: sub_dict.get(key) for key in key_lst} for sub_dict in lst_dict]
# using "dict.get('key')" in case key is not present in sub dicts in example 
>>> my_list
[{'TABLE_NAME': 'TEST1', 'COLUMN_NAME': 'TEST_COLUMN'}, {'TABLE_NAME': 'TEST1', 'COLUMN_NAME': 'TEST_COLUMN2'}, {'TABLE_NAME': 'TEST2', 'COLUMN_NAME': 'TEST_COLUMN1'}]

where lst_dict and key_lst are holding the values mentioned in the question.

The issue with your attempt is that you are not creating separate dict for the sub dicts in your list.