Cannot connect to MongoDB replicaSet cluster

Question

I set up a MongoDB cluster with Docker which uses replicaSet rs0 .

Checking the status of the cluster everything seems fine:

{
  set: 'rs0',
  date: ISODate("2022-12-20T21:00:46.972Z"),
  myState: 1,
  term: Long("1"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
    lastCommittedWallTime: ISODate("2022-12-20T21:00:40.112Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
    appliedOpTime: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
    durableOpTime: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
    lastAppliedWallTime: ISODate("2022-12-20T21:00:40.112Z"),
    lastDurableWallTime: ISODate("2022-12-20T21:00:40.112Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671570000, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate("2022-12-20T20:45:09.901Z"),
    electionTerm: Long("1"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671569099, i: 1 }), t: Long("-1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671569099, i: 1 }), t: Long("-1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-20T20:45:09.999Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-20T20:45:10.856Z")
  },
  members: [
    {
      _id: 0,
      name: 'mongo1:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 973,
      optime: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-20T21:00:40.000Z"),
      lastAppliedWallTime: ISODate("2022-12-20T21:00:40.112Z"),
      lastDurableWallTime: ISODate("2022-12-20T21:00:40.112Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671569109, i: 1 }),
      electionDate: ISODate("2022-12-20T20:45:09.000Z"),
      configVersion: 1,
      configTerm: 1,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'mongo2:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 947,
      optime: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
      optimeDurable: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-20T21:00:40.000Z"),
      optimeDurableDate: ISODate("2022-12-20T21:00:40.000Z"),
      lastAppliedWallTime: ISODate("2022-12-20T21:00:40.112Z"),
      lastDurableWallTime: ISODate("2022-12-20T21:00:40.112Z"),
      lastHeartbeat: ISODate("2022-12-20T21:00:45.203Z"),
      lastHeartbeatRecv: ISODate("2022-12-20T21:00:46.101Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mongo1:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 1
    },
    {
      _id: 2,
      name: 'mongo3:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 947,
      optime: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
      optimeDurable: { ts: Timestamp({ t: 1671570040, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-20T21:00:40.000Z"),
      optimeDurableDate: ISODate("2022-12-20T21:00:40.000Z"),
      lastAppliedWallTime: ISODate("2022-12-20T21:00:40.112Z"),
      lastDurableWallTime: ISODate("2022-12-20T21:00:40.112Z"),
      lastHeartbeat: ISODate("2022-12-20T21:00:45.203Z"),
      lastHeartbeatRecv: ISODate("2022-12-20T21:00:46.100Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mongo1:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 1
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671570040, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671570040, i: 1 })
}

I am using pymongo to connect to the cluster. If I use just the node name of the primary node and connect directly to that, everything works just fine:

client = MongoClient("mongodb://localhost:27017")
collection = client["db"]["test"]
collection.insert_one({"abc": "def"})
print(list(collection.find()))

However, from what I can tell, this is NOT how you connect to a MongoDB cluster. Instead of directly choosing the primary Node, one instead should provide all Nodes in the cluster plus the replicaSet:

client = MongoClient("mongodb://localhost:27017,localhost:27018,localhost:27019?replicaSet=rs0")
collection = client["db"]["test"]
collection.insert_one({"abc": "def"})  # Hangs and eventually times out

The error raised:

pymongo.errors.ServerSelectionTimeoutError: 
mongo1:27017: [Errno 8] nodename nor servname provided, or not known,
mongo3:27017: [Errno 8] nodename nor servname provided, or not known,
mongo2:27017: [Errno 8] nodename nor servname provided, or not known, 
Timeout: 30s, 
Topology Description: 
    <TopologyDescription 
        id: 63a2230d721313f82429a19e, 
        topology_type: ReplicaSetNoPrimary, 
        servers: [
            <ServerDescription ('mongo1', 27017) 
                server_type: Unknown, 
                rtt: None, 
                error=AutoReconnect('mongo1:27017: [Errno 8] nodename nor servname provided, or not known')>, 
            <ServerDescription ('mongo2', 27017) 
                server_type: Unknown, 
                rtt: None, 
                error=AutoReconnect('mongo2:27017: [Errno 8] nodename nor servname provided, or not known')>, 
            <ServerDescription ('mongo3', 27017) 
                server_type: Unknown, 
                rtt: None, 
                error=AutoReconnect('mongo3:27017: [Errno 8] nodename nor servname provided, or not known')>]>

I believe this might be because mongo1/2/3 are not actual hostnames but rather docker containers which are only valid hostnames within the Docker network, meaning, to access those hostnames I'd have to run my Python script within the same network.

I also tested if secondary nodes are reachable if I use localhost:27018 and indeed, they are. pymongo tells me that I cannot insert directly into a secondary node, which makes sense.

My docker-compose.yaml :

version: '3'

services:
  mongo1:
    image: mongo:latest
    container_name: mongo1
    command: mongod --replSet rs0 --bind_ip localhost,mongo1
    ports:
      - "27017:27017"
    networks:
      - mongoCluster
    volumes:
      - /mnt/workspace/mongodb-cluster/node1:/data/db
  mongo2:
    image: mongo:latest
    container_name: mongo2
    command: mongod --replSet rs0 --bind_ip localhost,mongo2
    ports:
      - "27018:27017"
    networks:
      - mongoCluster
    volumes:
      - /mnt/workspace/mongodb-cluster/node2:/data/db
  mongo3:
    image: mongo:latest
    container_name: mongo3
    command: mongod --replSet rs0 --bind_ip localhost,mongo3
    ports:
      - "27019:27017"
    networks:
      - mongoCluster
    volumes:
      - /mnt/workspace/mongodb-cluster/node3:/data/db

networks:
  mongoCluster:
    driver: bridge

I also tried to not use a dedicated network and instead give the docker containers access to the hostnetwork, thus being able to define the replicaSet with localhost:<PORT> . While I was able to bring up the cluster, I still get the same error when trying to use it from Python.

All that being said, my question is now how I need to change my Docker or pymongo configuration such that I can specify all nodes in the connection string.

Answer 1

You general you cannot mix hostnames and localhost alias. If one ReplicaSet member is localhost then all members needs to be localhost , they must differ only in the port.

According rs.status() the members are mongo1:27017,mongo2:27017,mongo3:27017 , thus the connection string must be accordingly

MongoClient("mongodb://mongo1:27017,mongo2:27017,mongo3:27017?replicaSet=rs0")

Or you use localhost when you initiate the ReplicaSet:

rs.initiate(
  {
    _id: "rs0",
    members: [
      { _id: 0, host: "localhost:27017" },
      { _id: 1, host: "localhost:27018" },
      { _id: 2, host: "localhost:27019" }
    ]
  }
)