将 JSON output 转换为 dataframe 表中的 ZA7F5F35426B92378173B

Question

I am using a REST api to get a JSON file. I want to convert this JSON file into a dataframe so that i can upload it to a database.

The format of the JSON looks like this:

{result:
{__campaign_id__:
{"campaign_id": __campaign_id__ //id of the campaign in platform
  "name": xxx, // Campaign name
  "creatives":{ // list of creatives in the campaign
   __creative_id__:{
     "creative_id":__creative_id__,  // id of the creative
     "name":xxx, //name of the creative
     "device":xxx, // 0- mobile | 1- Desktop | 2- Instream video | 3- Responsive
     "width": x, //height of the placement in px
     "height": y, //w idth of the placement in px
     "analytics": {
       __live_id__:{
                "dt":xxx, //date in unix timestamp
                "impressions":xxx, //number of tracked ads
                "viewable":xxx,  // number of viewable impressions
                "engagement":xxx, // number of impressions user interacted or viewed video at least 25%
                "engagement_novtr":xxx, // number of impressions user interacted
                "ctr_unique":xxx, //number of unique clicks (one click per one impression)
                "ctr": xxx,  // number of total clicks
                "dwell":xxx, // exposure time
                "videos":[
                    {
                        "dt": 1607558400, //date in unix timestamp. Same as above
                        "unique": xxx, //number of unique video views
                        "id": "Video 1", //name of the video
                        "vtr_0": xxx,   // number of impressions who started watching video 1
                        "vtr_25": xxx,  // number of impressions who watched first quartile of video 1
                        "vtr_50": xxx,  // number of impressions who watched second quartile of video 1
                        "vtr_75": xxx,  // number of impressions who watched third quartile of video 1
                        "vtr_100": xxx  // number of impressions who finished watching the video of video 1
                    }
                ]
            }
        }
    },
   },
  },
}
}

All i want to get out is the data is the most granular way possible, so like this (simplified):

campaignID   creativeID    device         analytics  
1           1            pc             1  
1           1            pc             2  
1           2            mobile         1 
1           2            pc             2  
2           4            pc             5  
2           4            mobile         6  
2           6            pc             7  
2           5            mobile         7   
3           8            pc             9    

And so on. Basically just split the data into new lines so that every new line represents the finest split of the data possible. If that makes sense.

My request looks like

nexdReponse = requests.get("myURL", headers=call_headers)
json_nexdData = json.loads(nexdReponse.text)

Now i have my JSON in a dictionary with the JSON heirarchy. So a dictionary of nested dictionaries i need to convert to a dataframe.

Then I've tried normalizing my data with pd.Dataframe.from_dict(pd.json_normalize(myData) but it doesn't do what i want.

Is there a simple way or a library i can use for this or something? Obviously i'm kind of new to this, so just trying to figure out how this works.

Answer 1

The json file you uploaded is invalid,so i'm posting a random valid one

{
    "status": {
        "code": 200,
        "message": "ok"
    },
    "pagination": {
        "page": 1,
        "count": 100,
        "total": 292
    },
    "products": [
    {
            "id": 143,
            "created_at": "2019-11-19T04:30:14.000Z",
            "updated_at": "2019-11-19T04:30:19.000Z",
            "blacklisted": false,
            "average_score": 4.76109,
            "total_reviews": 2051,
            "url": "https://go/kytatohbhi",
            "external_product_id": "123455",
            "name": "bolo azadi",
            "description": "kaisebhi",
            "product_specs": [
                {
                    "name": "shradha anusar",
                    "value": "dhania"
                }
            ],
            "category": {
                "id": 1,
                "name": "bandhkarobhai"
            },
            "products_group": {
                "id": 3518659,
                "display_name": "makkede"
            },
            "images": [
                {
                    "original": "https://haule",
                    "square": "https://bulle",
                    "facebook": "https://sulle",
                    "facebook_square": "https://lulle",
                    "kind": "image"
                }
            ]
        },
        {
            "id": 148,
            "created_at": "2019-11-19T04:30:14.000Z",
            "updated_at": "2019-11-19T04:30:19.000Z",
            "blacklisted": false,
            "average_score": 4.76109,
            "total_reviews": 2051,
            "url": "https://kytatohbhi",
            "external_product_id": "123455",
            "name": "kuch bhi....khuch bhi",
            "description": "kabhi alvida na kehna",
            "product_specs": [
                {
                    "name": "shradha anusar",
                    "value": "dhania"
                },
        {
                    "name": "swaad anusar",
                    "value": "namak"
                }
            ],
            "category": {
                "id": 1,
                "name": "bandhkarobhai"
            },
            "products_group": {
                "id": 3518659,
                "display_name": "makkede"
            },
            "images": [
                {
                    "original": "https://haule",
                    "square": "https://bulle",
                    "facebook": "https://sulle",
                    "facebook_square": "https://lulle",
                    "kind": "image"
                }
            ]
        }
    ]
}

once the json is read, you can save it a dataframe, note that key products is a list

df = pd.json_normalize(data['products'])

then you can use an explode function to split rest of the columns which are nested

def explode_node(child_df, column_value):
    child_df = child_df.dropna(subset=[column_value])
    if isinstance(child_df[str(column_value)].iloc[0], str):
        child_df[column_value] = child_df[str(column_value)].apply(ast.literal_eval)
    expanded_child_df = (pd.concat({i: json_normalize(x) for i, x in child_df.pop(str(column_value)).items()}).reset_index(level=1,drop=True).join(child_df, how='right', lsuffix='_left', rsuffix='_right').reset_index(drop=True))
    expanded_child_df.columns = map(str.lower, expanded_child_df.columns)
    return expanded_child_df