在python中快速制作一个大的json文件

Question

I have a multiple-json file of about 5 GB on which I do some exploratory data analysis. The problem is that each time I load the file it takes about 1 minute to load it when using this code:

with open(json_fn, 'r') as f:   # multiple jsons in one file! (one per line)
   for line in f:
       data = json.loads(line)

Is there a more efficient way to store these data for loading it in python? I was thinking about pickle (as it is a binary format which is usually faster) but it seems to be even slower . Any recommendations what I could use to avoid waiting 1 minute every time?

Answer 1

You can use ijson for this purpose. ijson allows reading the file lazily as a stream.

import ijson

json_data = ijson.parse(open(FILE_PATH, 'r'))

for prefix, event, value in json_data:
    print(value)

Refer this .

NOTE: With the help of backends mentioned in this post you can increase the performance quiet a lot.