包含列表数据的字典，基于列表中的值进行过滤

Question

I have test data which is gathered based on multiple inputs, and results in a single output. I'm currently storing this data in a dictionary whose keys are my parameter/ results labels, and whose values are the test conditions and results. I would like to be able to filter the data so I can generate plots based on isolated conditions.

In my example below, my test conditions would be 'a' and 'b', and the result of the experiment would be 'c'. I want to filter my data so I get a dictionary with the same key, value structure and only my filtered results. However my current dictionary comprehension returns an empty dictionary. Any advice to get the desired result?

Current Code:

data = {'a': [0, 1, 2, 0, 1, 2], 'b': [10, 10, 10, 20, 20, 20], 'c': [1.3, 1.9, 2.3, 2.3, 2.9, 3.4]}
filtered_data = {k:v for k,v in data.iteritems() if v in data['b'] >= 20}

Desired Result:

{'a': [0, 1, 2], 'b': [20, 20, 20], 'c': [2.3, 2.9, 3.4]}

Current Result:

{}

Also, is this dictionary of lists a good schema to store data of this type, given that I'm going to want to filter the results, or is there a better way to accomplish this?

Answer 1

Consider using the pandas module for this type of work.

import pandas as pd
df = pd.DataFrame(data)
df = df[df["b"] >= 20]
print(df)

It appears like this will give you what you want. You are using the dictionary key to represent the column name and the values are just rows in a given column, so it is amenable to using a dataframe.

Result:

   a   b    c
3  0  20  2.3
4  1  20  2.9
5  2  20  3.4

Answer 2

use this:

k:[v[i] for i,x in enumerate(v) if data['b'][i] >= 20] for k,v in data.items()}

Desired Result:

{'a': [0, 1, 2], 'c': [2.3, 2.9, 3.4], 'b': [20, 20, 20]}

Answer 3

Are all of the dictionary value lists in matching orders? If so, you could just look at whichever list you want to filter by, say 'b' in this case, find the values you want, and then either use those indices or the same slice on the other values in the dictionary.

For example:

matching_indices = []
for i in data['b']:
    if data['b'][i] >= 20:
        matching_indices.append(i)
new_dict = {}
for key in data:
    for item in matching_indices:
        new_dict[key] = data[key][item]

You could probably figure a dictionary comprehension for it if you wanted. Hopefully this is clear.

Answer 4

you can change this into a method which would give it more flexibility. Your current logic means that dataset a and c are neglected because there are no values greater than or equal to 20:

data = {'a': [0, 1, 2, 0, 1, 2], 'b': [10, 10, 10, 20, 20, 20], 'c': [1.3, 1.9, 2.3, 2.3, 2.9, 3.4]}
filter_vals = ['a', 'b']
new_d = {}
for k, v in data.iteritems():
  if k in filter_vals:
    new_d[k] = [i for i in v if i >= 20]

print new_d

Now i'm not a big fan if many if statements, but something like this is straight forward and can be called many times

def my_filter(operator, condition, filter_vals, my_dict):
  new_d = {}
  for k, v in my_dict.iteritems():
    if k in filter_vals:
      if operator == '>':
        new_d[k] = [i for i in v if i > condition]
      elif operator == '<':
        new_d[k] = [i for i in v if i < condition]
      elif operator == '<=':
        new_d[k] = [i for i in v if i <= condition]
      elif operator == '>=':
        new_d[k] = [i for i in v if i >= condition]
  return new_d

Answer 5

I agree with the pandas approach above.

If for some reason you hate pandas or are an old school computer scientist, tuples are a good way to tore relational data. In your example, the a, b, and c lists are columns rather than rows. For tuples, you would want to store the rows as:

data = {'a':(0,10,1.3),'b':(1,10,1.9),'c':(2,10,2.3),'d':(0,20,2.3),'e':(1,20,2.9),'f':(2,20,3.4)}

where the tuples are stored in the (condition1, condition2, outcome) format you described and you can call a single test or filter a set as you describe. From there you can get a filtered set of results as follows:

filtered_data = {k:v for k,v in data.iteritems() if v[1]>=20}

which returns:

{'d': (0, 20, 2.3), 'e': (1, 20, 2.9), 'f': (2, 20, 3.4)}