Facing issue with Mongoexport json file "_id" column

Question

I am exporting mongo collection to json format and then loading that data to bigquery table using bq load command.

mongoexport --uri mongo_uri  --collection coll_1 --type json --fields id,createdAt,updatedAt --out data1.csv 

The json row looks like below:

{"_id":{"$oid":"6234234345345234234sdfsf"},"id":1,"createdAt":"2021-05-11 04:15:15","updatedAt":null}

but when i run bq load command in bigquery it gives below error:

Invalid field name "$oid". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 300 characters long.

I think if mongoexport json contains {"_id": ObjectId(6234234345345234234sdfsf)}, my issue will be solved.

Is there any way to export json like this? Or any other way to achive this?

Note: i can't use csv format because mongo documents contain commas.

Answer 1

By default, _id holds an ObjectId value, so it's better to store data in {"_id": ObjectId(6234234345345234234sdfsf)} format instead of storing it in "_id":{"$oid":"6234234345345234234sdfsf"}.

As you mentioned if json contains {"_id": ObjectId(6234234345345234234sdfsf)} your problem will be solved.

Answer 2

Replace $oid with oid . I'm using Python, so the code below worked:

 with fileinput.FileInput("mongoexport_json.txt", inplace=True, encoding="utf8") as file:
 for line in file:
    print(line.replace('"$oid":', '"oid":'), end='')