How do i split out nested json string into 3 columns and relate it to the user_id column in a dataframe?

Question

I currently have a dataframe with 2 columns: user_id, items. Example data is:

user_id = 01e716c9bec1423e1

items = [{'item_id': '31499834785910', 'price': 3000.0, 'quantity': 2.0}, {'item_id': '31919169077366', 'price': 2500.0, 'quantity': 1.0}, {'item_id': '32130388426870', 'price': 5000.0, 'quantity': 1.0}, {'item_id': '22640717824118', 'price': 2000.0, 'quantity': 1.0}, {'item_id': '32044129157238', 'price': 3000.0, 'quantity': 1.0}, {'item_id': '31492182245494', 'price': 1500.0, 'quantity': 1.0}]

Items can contain more nested items,less or even none. What i want as an end product is:

df['user_id','item_id','price','quantity'] with obviously a row per item. 

So far i have tried:

import pandas as pd
import ast
import numpy as np
import pyodbc
import json

mylist = list(df['items'])
mynewlist = []
for l in mylist:
    mynewlist.append(ast.literal_eval(l))
data_items = pd.DataFrame(mynewlist)
data_new = pd.concat([df,data_items],axis=1)
del data_new['items']

but this just messes the entire dataframe up and adds about 40 columns on NaN and still doesnt break up the json.

I have found a few answers on this but none of them seem to help me out at all. so any help would be greatly appreciated. Also i have tried json_normalize and can't seem to figure it out.

I feel as thought is is a detailed question and apologies for not providing it in table format as i can't figure out how to do that, but if you need more info please let me know.

Answer 1

You can use a simple for loop to add the user_id key and value to each dictionary in the items list:

import pandas as pd

user_id = '01e716c9bec1423e1'

items = [{'item_id': '31499834785910', 'price': 3000.0, 'quantity': 2.0},
         {'item_id': '31919169077366', 'price': 2500.0, 'quantity': 1.0},
         {'item_id': '32130388426870', 'price': 5000.0, 'quantity': 1.0}, 
         {'item_id': '22640717824118', 'price': 2000.0, 'quantity': 1.0},
         {'item_id': '32044129157238', 'price': 3000.0, 'quantity': 1.0},
         {'item_id': '31492182245494', 'price': 1500.0, 'quantity': 1.0}]

# add the user_id to each dictionary
for item in items:
    item['user_id'] = user_id

df = pd.DataFrame(items)

print(df)

Output:

          item_id   price  quantity            user_id
0  31499834785910  3000.0       2.0  01e716c9bec1423e1
1  31919169077366  2500.0       1.0  01e716c9bec1423e1
2  32130388426870  5000.0       1.0  01e716c9bec1423e1
3  22640717824118  2000.0       1.0  01e716c9bec1423e1
4  32044129157238  3000.0       1.0  01e716c9bec1423e1
5  31492182245494  1500.0       1.0  01e716c9bec1423e1

Answer 2

An alternative without using a loop is:

import pandas as pd

user_id = ['01e716c9bec1423e1']

items = [{'item_id': '31499834785910', 'price': 3000.0, 'quantity': 2.0},
     {'item_id': '31919169077366', 'price': 2500.0, 'quantity': 1.0},
     {'item_id': '32130388426870', 'price': 5000.0, 'quantity': 1.0}, 
     {'item_id': '22640717824118', 'price': 2000.0, 'quantity': 1.0},
     {'item_id': '32044129157238', 'price': 3000.0, 'quantity': 1.0},
     {'item_id': '31492182245494', 'price': 1500.0, 'quantity': 1.0}]

df = pd.DataFrame(items)

# since user_id is a list, you just multiply by len(df) to have a list with the compatible length
df['user_id'] = user_id * len(df)