如何在 python 中以特定格式读取 JSON 文件作为 stream

Question

I have a huge file ~8 GB in JSON and I want to read it as stream with chunks of 1000 examples at a time. So I searched a lot and tried several packages but not of them really did the job.

The format of my file is as follows:

{
    "Elem1": [
       {
            "orgs": [],
       },
       {
           "people":[]
       },
    ],
   "Elem2"":[
       {
            "orgs": [],
       },
       {
           "people":[]
       },
    ],
...
}

As you can see, each element is saved as a tuple with two dicts and reoccurring keys in it. Is there a way how I could read/load/process the file above in chunks of elements ie chunk_1 = [ Elem1, Elem2, ... ] into the RAM and get the values for the keys out of them? Any ideas how to do that? Would appreciate your help.

Best regards Chris

Answer 1

As Serge said, you will need a custom parser to do the job. Something like below:

stack = []

json_string = ""
count = 0
with open(filename) as f:
  while True:
    c = f.read(1)
    if c == '{' or c == '[':
      stack.append(c)
    elif c == '}' or c == ']':
      stack.pop()
    json_string += c
    if len(stack) == 1:
      json_string += '}'
      count += 1
    if count == DESIRED_COUNT :
      break

The final json_string will contain the json with DESIRED_COUNT of objects