如何在 Python 中逐行读取巨大且有效的 JSON 文件

Question

I've been trying to use this code to read a huge JSON file (It contains 80+ million records) line by line:

import json
import pandas as pd


lines = []

with open('file_path','r') as f:
    for line in f:
            lines.append(json.loads(line))      
            df = pd.DataFrame(lines)

But this gives an error:

JSONDecodeError: Expecting property name enclosed in double quotes

Then, I used replace function with below code,

import json
import pandas as pd


lines = []
jstr = ""


with open('filepath','r') as f:
    for line in f:
            jstr = f'{jstr}{line}'
            jstr = line.replace("'", '"')
            lines.append(json.loads(jstr))
            df = pd.DataFrame(lines)

But I can only read first six rows and then I got this error:

JSONDecodeError: Expecting ',' delimiter

It is ensured that json is a valid format but I don't know what to do.

Would anyone help me how to handle this problem?

Answer 1

Maybe are you searching this?

from pandas as pd

df = pd.read_json('data/simple.json')