使用python将数据加载到mongodb atlas中

Question

I have tried to write a python script to load data into mongodb but it failing. I am using mongodb atlas. I am executing the script from ubuntu. It is able to establish connection. But never see the either the db being created nor the collections.

The code I have written:

from pymongo import MongoClient
import urllib
import sys
import pandas as pd
import pymongo
import json
import os

def import_content(filepath):

    mng_client = pymongo.MongoClient("mongodb://sathish1000:siva123.A@cluster0-shard-00-00-vgsfk.mongodb.net:27017/<DATABASE>?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin")

    mng_db = mng_client['local']
    collection_name = 'collection_name'
    db_cm = mng_db[collection_name]
    print(db_cm)
    cdir = os.path.dirname(__file__)
    file_res = os.path.join(cdir, filepath)
    data = pd.read_csv(file_res)
    data_json = json.loads(data.to_json(orient='records'))
    print(db_cm.find() )
    i = 0
    for data in data_json:
        i = i+1
        print(i)
        business = {"value":i}
        db_cm.insert_one(data)

if __name__ == "__main__":
    filepath = '/home/sathish/Downloads/Train.csv'
    import_content(filepath)

The error I am getting is as below:

Traceback (most recent call last):
  File "LoadMongo.py", line 32, in <module>
    import_content(filepath)
  File "LoadMongo.py", line 28, in import_content
    db_cm.insert_one(data)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 654, in insert_one
    with self._socket_for_writes() as sock_info:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 825, in _get_socket
    with server.get_socket(self.__all_credentials) as sock_info:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/server.py", line 168, in get_socket
    with self.pool.get_socket(all_credentials, checkout) as sock_info:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/pool.py", line 792, in get_socket
    sock_info.check_auth(all_credentials)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/pool.py", line 512, in check_auth
    auth.authenticate(credentials, self)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/auth.py", line 470, in authenticate
    auth_func(credentials, sock_info)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/auth.py", line 450, in _authenticate_default
    return _authenticate_scram_sha1(credentials, sock_info)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/auth.py", line 229, in _authenticate_scram_sha1
    res = sock_info.command(source, cmd)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/pool.py", line 424, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/pool.py", line 552, in _raise_connection_failure
    raise error
  File "/usr/local/lib/python3.5/dist-packages/pymongo/pool.py", line 419, in command
    collation=collation)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/network.py", line 110, in command
    response, codec_options=codec_options)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/helpers.py", line 128, in _unpack_response
    if error_object["$err"].startswith("not master"):
KeyError: '$err'

Can someone help me with this ?

Answer 1

This is because you're trying to write or insert data into the local database . The local database is used to store data for the replication process, and other instance-specific data. The local database is invisible to replication, resulting in collections in the local database are not replicated.

You should modify your code, as below example:

m_client = pymongo.MongoClient("mongodb://...")
m_db = m_client["database_name"]
m_collection = m_db["collection_name"]

An additional improvement that you can make is to use Bulk Write Operations instead of writing one document at a time. For example: if you have 1000 lines in your file, instead of sending 1000 insert_one() , you could send 1 bulk insert.