Iterate through list of dictionaries in python

Question

I have a list of dictionaries in python with the following form:

[{'item_value': 0.1, 'date': datetime.datetime(2017, ...), 'item_index': 1.0}, 
{'item_value': 0.22, 'date': datetime.datetime(2016, ...), 'item_index': 0.1}, 
{'item_value': 0.21, 'date': datetime.datetime(2016, ...), 'item_index': 1.0}
 ,..., 
{'item_value': 1.03, 'date': datetime.datetime(2016, ...), 'item_index': 1.0}]

Variable item_index takes values: [0.0, 0.1, 0.2, ..., 1.0] while variable item_value values between [-1, 1]. I want to construct a numpy vector which contain all possible item_index with the most recent item_value using the date (by omitting duplicates with the same item_value and keeping the most recent ones).

I am using the proposed solution:

np.array([d["item_value"] for d in sorted(my_list, key=lambda x: x["date"]))}

I create a numpy vector which contain all item_values sorted concerning the date [1.03, 0.22, 0.21, 0.1] in the case of the example. However, I want to return a vector like the following example:

[0, 0.22, 0, 0, 0, 0, 0, 0, 0, 0.1]

Each position of vector to represent the 11 possible values for item_index and have as a value the most recent value of the item_value. How can I do so?

EDIT

One example can be:

[{'item_value': 0.0, 'date': datetime.datetime(2017, 10, 11, 13, 39, 36, 979000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2017, 10, 11, 13, 40, 2, 368000), 'item_index': 1.0}
{'item_value': -1.0, 'date': datetime.datetime(2017, 10, 23, 9, 35, 20, 741000), 'item_index': 1.0}
{'item_value': -1.0, 'date': datetime.datetime(2017, 10, 23, 9, 35, 41, 915000), 'item_index': 0.8}
{'item_value': 0.0, 'date': datetime.datetime(2017, 10, 23, 9, 36, 2, 763000), 'item_index': 0.5}
{'item_value': 0.0, 'date': datetime.datetime(2017, 10, 23, 11, 40, 22, 427000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2017, 11, 14, 7, 33, 9, 131000), 'item_index': 1.0}
{'item_value': 0.51, 'date': datetime.datetime(2017, 11, 15, 12, 50, 25, 14000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2018, 1, 19, 14, 15, 46, 761000), 'item_index': 1.0}
{'item_value': -0.49, 'date': datetime.datetime(2018, 1, 19, 14, 16, 30, 207000), 'item_index': 1.0}
{'item_value': -0.009000000000000005, 'timestamp': datetime.datetime(2018, 1, 19, 16, 32, 30, 631000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2018, 1, 19, 16, 33, 19, 509000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2018, 1, 19, 16, 44, 59, 483000), 'item_index': 1.0}
{'item_value': -0.33299999999999996, 'date': datetime.datetime(2018, 1, 19, 18, 13, 17, 67000), 'item_index': 1.0}
{'item_value': 1.0, 'date': datetime.datetime(2018, 1, 19, 18, 13, 48, 443000), 'item_index': 1.0}
{'item_value': -0.33299999999999996, 'date': datetime.datetime(2018, 1, 19, 18, 14, 22, 871000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2018, 1, 28, 11, 45, 48, 223000), 'item_index': 1.0}
{'item_value': 0.005000000000000003, 'timestamp': datetime.datetime(2018, 1, 28, 11, 46, 7, 481000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2018, 1, 28, 11, 46, 27, 845000), 'item_index': 1.0}
{'item_value': 0.0, 'date': datetime.datetime(2018, 1, 28, 11, 46, 50, 386000), 'item_index': 1.0}]

Answer 1

A oneliner could be as follows:

indexes = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

my_filtered_lists = [sorted([d for d in my_list if d['item_index'] == i], 
                            key=lambda x: x["date"])
                        for i in indexes ]

result = [l[-1]['item_value'] if len(l)>0 else 0  for l in my_filtered_lists]

For each index you filter the list, and sort each filtered list as desired and get the item_value of the last element. If the data set is big enough, this could be a little memory demanding, since you are creating one extra list for each item_idex .

Tested with:

 my_list = [
{'item_value': 0.1, 'date': datetime.datetime(2017, 05, 01), 'item_index': 1.0}, 
{'item_value': 0.22, 'date': datetime.datetime(2016,05,01), 'item_index': 0.1}, 
{'item_value': 0.21, 'date': datetime.datetime(2017, 05, 01), 'item_index': 0.1},
{'item_value': 1.03, 'date': datetime.datetime(2016,05,01), 'item_index': 1.0}]

It returns: [0, 0.21, 0, 0, 0, 0, 0, 0, 0, 0, 0.1] wich I understand is the expected output.

Answer 2

One solution would be to create an intermediate dict key -> value only keeping the most up-to-date values:

d = dict()
for value in sorted(my_list, key=lambda x: x["date"]):
   d[value['item_index']] = d[value['item_value']]

Another solution would be to convert the list to a pandas DataFrame , sort by date, group by item_index with the last() function to only keep the latest record of the dataframe.