Reading dictionary stored on text file and convert to pandas dataframe [duplicate]

Question

: ：

Closed last year .

Pandas read nested json (1 answer)

I have a text file that contains a series of data in the form of dictionary. I would like to read and store as a data frame in pandas. How would I read. I read pd.csv yet it does not give me the dataframe. Can anyone help me with that? You can download the text file Here

Thanks,

Zep,

Answer 1

You can convert you data to json after reading it as string, then use pandas.read_json() to convert your json to a dataframe.

Example:

import json
from pandas.io.json import json_normalize

f = open("file.txt", "w+")
contents = f.read()
contents = contents.replace("\n", "")
json_data = json.loads(contents)
df = json_normalize(json.loads(data))

You should have your data as a dataframe after that.

Hope this helps!

Answer 2

The problem is you have a nested json. Try using json_normalize instead:

import requests     #<-- requests library helps us handle http-requests
import pandas as pd

id_ = '1DbfQxBJKHvWO2YlKZCmeIN4al3xG8Wq5'
url = 'https://drive.google.com/uc?authuser=0&id={}&export=download'.format(id_)
r = requests.get(url)

df = pd.io.json.json_normalize(r.json())
print(df.columns)

---

or from hard drive, and json_normalize as wants to read a dictionary object and not a path:

import pandas as pd
import json

with open('myfile.json') as f:
    jsonstr = json.load(f)

df = pd.io.json.json_normalize(jsonstr)

Returns:

Index(['average.accelerations', 'average.aerialDuels', 'average.assists',
       'average.attackingActions', 'average.backPasses', 'average.ballLosses',
       'average.ballRecoveries', 'average.corners', 'average.crosses',
       'average.dangerousOpponentHalfRecoveries',
       ...
       'total.successfulLongPasses', 'total.successfulPasses',
       'total.successfulPassesToFinalThird', 'total.successfulPenalties',
       'total.successfulSmartPasses', 'total.successfulThroughPasses',
       'total.successfulVerticalPasses', 'total.throughPasses',
       'total.verticalPasses', 'total.yellowCards'],
      dtype='object', length=171)

---

Another idea would be to store the nested objects in a Series (and you can let a dictionary hold that those series).

dfs = {k: pd.Series(v) for k,v in r.json().items()}    
print(dfs.keys()) 
# ['average', 'seasonId', 'competitionId', 'positions', 'total', 'playerId', 'percent'])
print(dfs['percent'])

Returns:

aerialDuelsWon                   23.080
defensiveDuelsWon                18.420
directFreeKicksOnTarget           0.000
duelsWon                         33.470
fieldAerialDuelsWon              23.080
goalConversion                   22.581
headShotsOnTarget                 0.000
offensiveDuelsWon                37.250
penaltiesConversion               0.000
shotsOnTarget                    41.940
...
yellowCardsPerFoul               12.500
dtype: float64

The data only has one entry though.