Slowness in PyMongo Flask Application

Question

I'm developing a basic flask application to receive an input request from the user and insert that into MongoDB Atlas Cluster.

I have a route /save which is of type POST. This endpoint receives the request, makes a new mongo connection, inserts to mongo and finally closes the connection. This approach is slow , with average response latency of 700-800 ms even though I am only trying to insert one document.

Note-: My use case does not make sense to use bulk-insert.

Sample Code

app = Flask(__name__)
app.logger.setLevel(logging.INFO)

DBNAME = 'DBNAME as String'
CONNSTRING = 'CONNECTION as String'

class mongoDB:
    def __init__(self):
        try:
            self.client = MongoClient(CONNSTRING, maxPoolSize=None)
            self.database = self.client[DBNAME]
            app.logger.info('Mongo Connection Established')
        except Exception as e:
            app.logger.warning('Mongo Connection could not be established')
            app.logger.warning('Error Message: ' + str(e))

    def close_connection(self):
        try:
            self.client.close()
        except Exception as e:
            app.logger.warning('connection failed to close')
            app.logger.warning('Error Message: ' + str(e))

@app.route('/save', methods=['POST'])
def save():
    data_info = flask.request.get_json()
    
    try:
        db = mongoDB()
        image_collection = db.database['DUMMY_COLLECTION']
        image_collection.insert_one({'VALUE_ID' : data_info['value1'], 'VALUE_STRING' : data_info['value2']})
        app.logger.info('Inserted Successfully')
        return {'message': 'Success'}, 200, {'Content-Type': 'application/json'}
    except Exception as e:
        app.logger.error('Error Adding data to Mongo: ' + str(e))
        return {'message': 'Error'}, 500, {'Content-Type': 'application/json'}
    finally:
        db.close_connection()
        app.logger.info('connection closed')

if __name__ == '__main__':
    app.run()

However if I establish the Mongo connection at time of application initialization and keep it open and never close the connection the latency drops to 70-80ms.

Could someone please help understand the consequences of keeping an open connection, instead of establishing a new connection with each request? Or is there any method to reduce latency with multiple connections open?

Note-: Keeping multiple connections approach, I tried using writeConcern=0, maxPoolSize=None, and journal=False but all these did not improve the latency much at all.

Any help would be appreciated. Thanks

Answer 1

The MongoClient(CONNSTRING, maxPoolSize=None) is not just a single connection but a connection pool. Meaning that already with that object you can have multiple concurrent requests to MongoDB. By setting maxPoolSize=None you make them limitless (which can have some implications under heavy load).

It is an antipattern to create a connection pool per request (as you realized by the high latency) the reason for that is that you need to pay each time the cost to create the connection pool and the handshake to the database.

The best way is to initiate one on startup and then maintain it. Meaning that you should handle all the exceptions that might arise in case of DB or network failures. However, I assume that most things are already handled by MongoClient already.