I'm using an instagram scraper that outputs a multi-row JSON file and I'd like to select certain values from that file and assign them to a DataFrame.
When I try to use panda's pd.read_json it only saves the first level to each dataframe.
As an example, I'd like to have a dataframe with the first row that contains (JSON variable in parenthesis):
Likes ("edge_media_preview_like": {"count": 1356...)
Comment Count ("edge_media_to_comment": {"count": 44)
The JSON file looks like this:
{
"GraphImages": [
{
"__typename": "GraphImage",
"comments_disabled": false,
"dimensions": {
"height": 770,
"width": 1080
},
"display_url": "https:abc123.com",
"edge_media_preview_like": {
"count": 1356
},
"edge_media_to_caption": {
"edges": [
{
"node": {
"text": "TEXT EXAMPLE 123"
}
}
]
},
"edge_media_to_comment": {
"count": 44
},
"gating_info": null,
"id": "2219687023504340370",
"is_video": false,
"media_preview": "abc123media",
"owner": {
"id": "212343915"
},
"shortcode": "B7N6ZZkhTWS",
"tags": [],
"taken_at_timestamp": 1578827334,
"thumbnail_resources": [
{
"config_height": 150,
"config_width": 150,
"src": "abc123.com"
},
{
"config_height": 240,
"config_width": 240,
"src": "abc123.com"
},
{
"config_height": 320,
"config_width": 320,
"src": "https://abc123.com"
},
{
"config_height": 480,
"config_width": 480,
"src": "https:/abc123.com"
},
{
"config_height": 640,
"config_width": 640,
"src": "https://abc123.com"
}
],
"thumbnail_src": "https://abc123.com",
"urls": [
"https://abc123.com"
],
"username": "abc123"
}
]
}
I'm looking for:
ImageNumber Likes CommentCount
0 1 1356 44
1 ... ... ...
Thank you!
Adding wrong result when using pd.read_json:
GraphImages
0 {'__typename': 'GraphImage', 'comments_disable...
1 {'__typename': 'GraphImage', 'comments_disable...
2 {'__typename': 'GraphImage', 'comments_disable...
3 {'__typename': 'GraphImage', 'comments_disable...
The following should work,
import json
with open('ig.json') as json_file:
dct = json.load(json_file)
df = pd.io.json.json_normalize(dct, record_path="GraphImages")[["edge_media_preview_like.count", "edge_media_to_comment.count"]].rename({"edge_media_preview_like.count":"Likes", "edge_media_to_comment.count": "CommentCount"}, axis=1)
df["ImageNumber"] = df.index + 1
Which produces,
Likes CommentCount ImageNumber
0 1356 44 1
I'm not sure where the ImageNumber
is coming from. But I assume it's the order of items appear in GraphImages
. If so, df.Index + 1
would give you that.
I found the answer. Turns out that this instagram-scraper outputs a JSON file that is composed of a dictionary, within a list, within a dictionary. The code to extract is as follows:
import json
import pandas as pd
with open('ig.json') as json_file:
data = json.load(json_file)
data['Likes'] = data['GraphImages'][0]['edge_media_preview_like']['count']
...
I hope that can help people in the future!
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.