在Python中使用PyMongo在MongoDB中插入文档

Question

I am inserting documents in MongoDB using the PyMongo library in Python. The pandas dataframe has 37 fields and 60k records (link to dataset: https://drive.google.com/open?id=119T4uhvHc7CAwJgZRselWXpstAQhkj90 ). All fields in the dataframe have been converted to str type. I am getting the following error:

OverflowError: MongoDB can only handle up to 8-byte ints

The error still persists when I insert chunks of 2500 documents using a for loop.

Code snippet:

import pandas as pd
import pymongo

client = pymongo.MongoClient()
db = client['patenting_in_psi']
collection = db['sample5']

df=pd.read_excel(r"C:\Users\mazin\1-601.xlsx")

collection.insert_many((df.to_dict('records')))

Answer 1

Some fields with missing data need to be normalized before converting the dataframe to a dictionary.

Values for DWPIAccessionNumber have to be normalized. For example, in record number 2524 is a 64-bit integer whose value is 20100000000000001078890512051682672902079220850980264522702989250781417512482524046970942628331980756243236345447307144055181790035144112662138043858072629370129477827567049201927634798584141270252235498775249725404749823022689297835494826055102466304887343437187655164225642338109880434082104977849399115776 . This may be converted to a bson.int64.Int64 type or conveniently typed as str ( there are instances where this value is a str - see record number 23 or a nan . )

df['DWPIAccessionNumber'] = df['DWPIAccessionNumber'].astype(str)

Also PublicationDate field needs to be normalized as well. For example, in record number 24696 its value is missing. You either drop the field, set some date or fill it a zero.

df['PublicationDate'].fillna(0, inplace=True)

Now, your data is ready to be converted to a dictionary then inserted.