How to get all values from a list of dict that matches key prefix?

Question

I have a list of dictionaries, for example:

[
 {
  'id': '11110110011',
  'a_dept_performance': 3,
  'a_group_performance': 2,
  'a_user_performance': 3,
  'f_service': 4,
  'f_facility_service': 2,
  'k_helpful': 2
  ...
 },
 {
  'id': '11110110012',
  'a_dept_performance': 3,
  'a_group_performance': 3,
  'a_user_performance': 3,
  'f_service': 2,
  'f_facility_service': 3,
  'k_helpful': 2
  ...
 },
 ...
]

I want to retrieve all value that matches with a specific prefix, like:

a_ = [3, 2, 3, 3, 3, 3, ...]
f_ = [4, 2, 2, 3, ...]
k_ = [2, 2, ...]

Any way to implement this? My data is big, so I'm wondering if there's a faster way to do it.

Answer 1

you can use collections.defaultdict to store your values:

from collections import defaultdict

my_prefixes = {'a_', 'f_', 'k_'}
result = defaultdict(list)
for d in my_list:
    for k, v in d.items():
        if k[:2] in my_prefixes:
            result[k[:2]].append(v)
print(result)

output:

defaultdict(<class 'list'>, {'a_': [3, 2, 3, 3, 3, 3], 'f_': [4, 2, 2, 3], 'k_': [2, 2]})

if you know that all the key are the same for every dictionary you can bring a plus of speed as @golf_cy was mentioned:

from collections import defaultdict
my_prefixes = {'a_' : ['a_dept_performance', 'a_group_performance'],
               'f_': ['f_service', 'f_facility_service'],
               'k_': ['k_helpful']}

result = defaultdict(list)
for d in my_list:
    for k, v in my_prefixes.items():
        result[k].extend([d[e] for e in v])

Answer 2

I can't say if there is a faster way to do this, but perhaps you should think of changing your data structure if your data is large. Your setup does not seem like a particularly efficient or intuitive method to store your data.

Consider a database, or if you want to stay within Python, look at Pandas.