Create keys for dictionary based on length of list of lists

Question

I have the following list:

my_list = [[['pd', 1],
           ['pd_de', None],
           ['pd_amnt', '$10.00']],
           [['pd', 1],
           ['pd_de', '5/1/19 '],
           ['pd_amnt', '$100.00 '],
           ['pd', 2],
           ['pd_de', '5/1/20 '],
           ['pd_amnt', '$200.00 ']],
           [['pd', 1],
           ['pd_de', None],
           ['pd_amnt', None]],
           [['pd', 1],
           ['pd_de', '5/1/19 '],
           ['pd_amnt', '$300.00 '],
           ['pd', 2],
           ['pd_de', '5/1/20 '],
           ['pd_amnt', '$600.00 '],
           ['pd', 3],
           ['pd_de', '6/1/18'],
           ['pd_amnt', '$450.00']]]

Using this, I would like to create a list of dictionaries. I am dong the following to create a list of dictionaries,

list_dict = []

for i in my_list:
    temp_dict = {}
    for j in i:
        temp_dict[j[0]] = j[1]
    list_dict.append(temp_dict)

And I am getting an output like this, which I do not want,

[{'pd': 1, 'pd_de': None, 'pd_amnt': '$10.00'},
 {'pd': 2, 'pd_de': '5/1/20 ', 'pd_amnt': '$200.00 '},
 {'pd': 1, 'pd_de': None, 'pd_amnt': None},
 {'pd': 3, 'pd_de': '6/1/18', 'pd_amnt': '$450.00'}]

I need an output like this,

[{'pd_1': 1, 'pd_de_1': None, 'pd_amnt_1': '$10.00'},
 {'pd_1': 1, 'pd_de_1': '5/1/19', 'pd_amnt_1': '$100.00', 'pd_2': 2, 'pd_de_2': '5/1/20 ', 'pd_amnt_2': '$200.00 '},
 {'pd_1': 1, 'pd_de_1': None, 'pd_amnt_1': None},
 {'pd_1': 1, 'pd_de_1': '5/1/19', 'pd_amnt_1': '$300.00','pd_2': 2, 'pd_de_2': '5/1/20', 'pd_amnt': '$600.00','pd_3': 1, 'pd_de_3': '6/1/18', 'pd_amnt_3': '$450.00'}]

If you see above, they are okay when the list inside has a length of 3. If it is more than 3, then it does not give the correct result.

I am also not sure how to create "_" in keys (ie 'pd_1') when I create keys for dictionary.

How can I achieve my desired output?

(Note: Not sure how to name the title, I said length of list, I might be wrong there because I am not familiar with pythonic terms)

Answer 1

Retaining the order of items:

import pandas as pd
from collections import OrderedDict

# my_list = ...

res = []
for l1 in my_list:
    d = OrderedDict()
    for l2 in l1:
        if l2[0] == 'pd':
            sfx = l2[1]
        d[f'{l2[0]}_{sfx}'] = l2[1].strip() if isinstance(l2[1], str) else l2[1]
    res.append(d)

df = pd.DataFrame(res)
print(df)

The output:

   pd_1 pd_de_1 pd_amnt_1  pd_2 pd_de_2 pd_amnt_2  pd_3 pd_de_3 pd_amnt_3
0     1    None    $10.00   NaN     NaN       NaN   NaN     NaN       NaN
1     1  5/1/19   $100.00   2.0  5/1/20   $200.00   NaN     NaN       NaN
2     1    None      None   NaN     NaN       NaN   NaN     NaN       NaN
3     1  5/1/19   $300.00   2.0  5/1/20   $600.00   3.0  6/1/18   $450.00

Answer 2

You can use additional variable ( counter ) to find key "index" which doesn't exist in dictionary yet:

result = []
for sub_list in my_list:
    temp = {}
    for key, value in sub_list:
        counter = 1
        while f"{key}_{counter}" in temp:
            counter  += 1
        temp[f"{key}_{counter}"] = value
    result.append(temp)

A bit more efficient solution will be to store counters into dict and increment them once key used:

result = []
for sub_list in my_list:
    counters = {}
    temp = {}
    for key, value in sub_list:
        if key in counters:
            counters[key] += 1
        else:
            counters[key] = 1
        temp[f"{key}_{counters[key]}" ] = value
    result.append(temp)

Usingcollections.defaultdict you can write it a bit shorter:

from collections import defaultdict

result = []
for sub_list in my_list:
    counters = defaultdict(int)
    temp = {}
    for key, value in sub_list:
        counters[key] += 1
        temp[f"{key}_{counters[key]}"] = value
    result.append(temp)

Answer 3

• I found a really cool way to do this.
• you can use defaultdict to increment key everytime you see it. And then add it your result dictionary.

list_dict = []

from collections import defaultdict

for i in my_list:
    temp_dict = {}
    incr = defaultdict(int)
    for j in i:
        incr[j[0]] += 1
        temp_dict[j[0] + '_' + str(incr[j[0]])] = j[1]
    list_dict.append(temp_dict)

Output:

[{'pd_1': 1, 'pd_de_1': None, 'pd_amnt_1': '$10.00'},
 {'pd_1': 1,
  'pd_de_1': '5/1/19 ',
  'pd_amnt_1': '$100.00 ',
  'pd_2': 2,
  'pd_de_2': '5/1/20 ',
  'pd_amnt_2': '$200.00 '},
 {'pd_1': 1, 'pd_de_1': None, 'pd_amnt_1': None},
 {'pd_1': 1,
  'pd_de_1': '5/1/19 ',
  'pd_amnt_1': '$300.00 ',
  'pd_2': 2,
  'pd_de_2': '5/1/20 ',
  'pd_amnt_2': '$600.00 ',
  'pd_3': 1,
  'pd_de_3': '6/1/18',
  'pd_amnt_3': '$450.00'}]

Answer 4

The reason you are getting this is because when you set a key in a dictionary to something, it will override any previous data. For example, you have this dictionary x = ["a":1, "b":2, "c":3] if you do x["d"] = 4 it will then be ["a":1, "b":2, "c":3, "d":4] but if you then do x["a"] = 3 it will be ["a":3, "b":2, "c":3, "d":4] .
The solution for you is to add each item into the dictionary with a number after the tag to represent which tag it is.

list_dict = []

for i in my_list:
    temp_dict = {}
    for j in i:
        a = 1
        while j[0]+"_"+str(a) in temp_dict:
            a += 1
        temp_dict[j[0]+"_"+str(a)] = j[1]
    list_dict.append(temp_dict)