How can I read multiple in json files in in folder structure?

Question

I'm trying to read Training data for machine learning in json files, but they are stored in nested folders.

I'd like to know how can I read each json file into pandas frame.

Answer 1

Imagine there are three JSON files in a nested folder under data folder.

$ tree data
data
├── date1
│   ├── date2
│   │   └── file1_date2.json
│   └── file1.json
└── file1.json

glob2 module can be used to fetch the JSON files recursively. glob returns a list of files.

from glob2 import glob
jsonFiles = glob('data/**/*.json') #Can be used absolute or relative paths
print(jsonFiles)

['data/file1.json',  'data/date1/file1.json',  'data/date1/date2/file1_date2.json']

JSON files ban be loaded into dataframe by iterating thru the list jsonFiles .

dfList = []
for jsonFile in jsonFiles:
    df = pd.read_json(jsonFile)
    dfList.append(df)
    
dfTrainingDF = pd.concat(dfList, axis=0)