Create a pandas dataframe from a python dictionary. Python dictionary has multiple keys and its values have both string and list data types

Question

I have a program that generate multiple dictionaries at each iteration, that follows this format.

# dict1
{"key1": "abc", "key2": True, "key3": []}
# dict2
{"key1": "def", "key2": False, "key3": ["a", "b", "c"]}
...

I would like to add these to a pandas dataframe.

On first iteration I create a df,

mydf = pd.DataFrame(data=dict1, columns=dict1.keys(), index=["abc"])

On subsequent iterations I append to mdf,

tdf = pd.DataFrame(dict2, columns=dict2.keys(), index=["def"])
mydf = mydf.append(tdf)

This approach works when there are no list values in my dictionaries. If there are list values(such as key3) I get the following ValueError,

ValueError: could not broadcast input array from shape (3) into shape (1)

Expected output,

    key1 key2 key3
abc  "abc"  True  []
def  "def"  False  ["a", "b", "c"]

Answer 1

You can first load your dict to a Series and then convert it to a DataFrame, keep appending to a df list, finally use pd.concat to merge your DFs.

df_list = []
df_list.append(pd.Series(dict1).to_frame().T)
df_list.append(pd.Series(dict2).to_frame().T)
pd.concat(df_list).set_index('key1', drop=False).rename_axis('')


    key1    key2    key3
abc abc     True    []
def def     True    [a, b, c]