合并具有不均匀字段的 JSON 文件的问题

Question

I am trying to merge multiple JSON files of some size , and uneven fields.

Here is what i mean:

JSON Example#1

{"firstName":"John", "lastName":"Doe"}

JSON Example#2

{"firstName":"John", "lastName":"Doe", "middleName":"Doe"}

Both the files are saved in the same location as the Merger.py that i am using merge these 2 JSON files.

Merger.py

import json
import glob

result = []
for f in glob.glob("*.json"):
    with open(f, "rb") as infile:
        result.append(json.load(infile))

with open("merged_file.json", "wb") as outfile:
     json.dump(result, outfile)

When i try to execute it however, i keep getting this error:

Traceback (most recent call last): File "Merger.py", line 10, in json.dump(result, outfile) File "C:\\Users\\<...Directory Path...>\\Programs\\Python\\Python37-32\\lib\\json__init__.py", line 180, in dump fp.write(chunk) TypeError: a bytes-like object is required, not 'str'

I understand this is due to uneven fields in the JSON files.

My question is, is there a work around for this situation ?

The original files are consisting of 91+ Million records each, so manual merging is out of the question (not that i have not tried that too) .

Answer 1

I understand this is due to uneven fields in the JSON files.

No,this is not problem.

You use with open(f, "rb") as infile and open("merged_file.json", "wb") . That is incorrect. You should use with open(f, "r") as infile and open("merged_file.json", "w") .

Only when we open a binary file instead of a text file,we use wb or rb .

TypeError: a bytes-like object is required, not 'str'

The exception has shown your problem.

So you code maybe should be this:

import json

result = []
for f in glob.glob("*.json"):
    with open(f, "r") as infile:
        result.append(json.load(infile))

Yourjson = {
    "result":result
}

with open("merged_file.json", "w") as outfile:
    json.dump(Yourjson, outfile,indent=4)