如何将多行 JSON 文件转换为 Dataframe

Question

I'm using an instagram scraper that outputs a multi-row JSON file and I'd like to select certain values from that file and assign them to a DataFrame.

When I try to use panda's pd.read_json it only saves the first level to each dataframe.

As an example, I'd like to have a dataframe with the first row that contains (JSON variable in parenthesis):

• Likes ("edge_media_preview_like": {"count": 1356...)

• Comment Count ("edge_media_to_comment": {"count": 44)

The JSON file looks like this:

{
    "GraphImages": [
        {
            "__typename": "GraphImage",
            "comments_disabled": false,
            "dimensions": {
                "height": 770,
                "width": 1080
            },
            "display_url": "https:abc123.com",
            "edge_media_preview_like": {
                "count": 1356
            },
            "edge_media_to_caption": {
                "edges": [
                    {
                        "node": {
                            "text": "TEXT EXAMPLE 123"
                        }
                    }
                ]
            },
            "edge_media_to_comment": {
                "count": 44
            },
            "gating_info": null,
            "id": "2219687023504340370",
            "is_video": false,
            "media_preview": "abc123media",
            "owner": {
                "id": "212343915"
            },
            "shortcode": "B7N6ZZkhTWS",
            "tags": [],
            "taken_at_timestamp": 1578827334,
            "thumbnail_resources": [
                {
                    "config_height": 150,
                    "config_width": 150,
                    "src": "abc123.com"
                },
                {
                    "config_height": 240,
                    "config_width": 240,
                    "src": "abc123.com"
                },
                {
                    "config_height": 320,
                    "config_width": 320,
                    "src": "https://abc123.com"
                },
                {
                    "config_height": 480,
                    "config_width": 480,
                    "src": "https:/abc123.com"
                },
                {
                    "config_height": 640,
                    "config_width": 640,
                    "src": "https://abc123.com"
                }
            ],
            "thumbnail_src": "https://abc123.com",
            "urls": [
                "https://abc123.com"
            ],
            "username": "abc123"
        }
    ]
}

I'm looking for:

    ImageNumber Likes   CommentCount
0   1           1356    44
1   ...         ...     ...

Thank you!

Adding wrong result when using pd.read_json:

    GraphImages
0   {'__typename': 'GraphImage', 'comments_disable...
1   {'__typename': 'GraphImage', 'comments_disable...
2   {'__typename': 'GraphImage', 'comments_disable...
3   {'__typename': 'GraphImage', 'comments_disable...

Answer 1

The following should work,

import json
with open('ig.json') as json_file:
    dct = json.load(json_file)

df = pd.io.json.json_normalize(dct, record_path="GraphImages")[["edge_media_preview_like.count", "edge_media_to_comment.count"]].rename({"edge_media_preview_like.count":"Likes", "edge_media_to_comment.count": "CommentCount"}, axis=1)
df["ImageNumber"] = df.index + 1

Which produces,

    Likes   CommentCount    ImageNumber
0   1356    44              1

I'm not sure where the ImageNumber is coming from. But I assume it's the order of items appear in GraphImages . If so, df.Index + 1 would give you that.

Answer 2

I found the answer. Turns out that this instagram-scraper outputs a JSON file that is composed of a dictionary, within a list, within a dictionary. The code to extract is as follows:

import json
import pandas as pd

with open('ig.json') as json_file:
    data = json.load(json_file)

data['Likes'] = data['GraphImages'][0]['edge_media_preview_like']['count']
...

I hope that can help people in the future!