MongoDB Performance Issue at AWS

Question

I have a few questions about MongoDB performance. Thanks in advance for any help.

DB Architecture

The DB has been installed according to instructions at: https://docs.aws.amazon.com/quickstart/latest/mongodb/step2.html

It has one bastion and three replicas.

The DB is used in a Ruby on Rails 5 app with MongoDB 4.09

Questions

1. I would like to know if the 'hosts' parameter should include all the replicas or only the primary?

mongoid.yml config

"mongoid": {
                "production": {
                    "clients": {
                        "default": {
                            "options": {
                                "user": "USER",
                                "password": "PASS****",
                                "auth_source": "admin",
                                "read": {
                                    "mode": "primary_preferred"
                                },
                                "max_pool_size": 200,
                                "min_pool_size": 10,
                                "ssl_verify": false
                            },
                            "database": "pro_db",
                            "hosts": [
                                "PRIMARY_REPLICANODE.3.0.53:27017",
                                "SECONDARY_NODE0.3.62.61:27017",
                                "SECONDARY_NODE1.3.80.80:27017"
                            ]
                        }
                    }
                }
            },

2. What is a good value for max_pool_size? Each Mongo DB node has around 16 GiB of RAM and 2 Virtual CPUS. Are there any other relevant parameters that should be modified from the default?

3. There is a performance issue on concurrent API requests with different parameters. API requests with the same parameter have an average response of 11 milliseconds, but as soon as a parameter changes and is concurrent the average is 8904 milliseconds. And some timeout. I would like to know what might cause the issue?

API Requests With Same parameter 11 ms

API Requests With Different parameters 8904 ms

Answer 1

1. All MongoDB hosts should be listed in application configuration. Which server is the primary one changes over time, for example during maintenance. If the application references only one server and this server is restarting, the application will have no way of locating any of the other servers (one of which would likely be the current primary) and may experience an outage.

   The driver performs discovery of nodes in the cluster and, if it is able to contact nodes in the seed list (this is the list of servers in the configuration), will then be able to find out about all of the other nodes in the cluster and follow primary changes. But if you happen to restart the application and at that time it is referencing only one server and that server is down, the application will not be able to find the other servers.

2. Max pool size is the maximum number of open connections that each client will keep around. A 16 GB RAM node should have no issue with 200 open connections. If max pool size is set too low, the application may experience errors as it will be unable to obtain a connection to perform operations.

   Note that the max of 200 roughly states that you are expecting to perform about 200 operations concurrently, which - for a typical Rails application - seems unlikely. But, there is nothing wrong with leaving the max at 200.

3. The specific performance question is not really answerable given the details you provided. Narrow down whether the time is spent in the application or in the database; if it's spent in the database, what the queries are and how they are different. If the time is spent in the application you generally would have to profile the application yourself as there are many possibilities where the time may be going.