mongodb replica set error "...replSetHeartbeat requires authentication..." after running rs.initiate()

Question

I have 2 virtual machines with mongodb running on both. I have created a user and roles and added data for testing. Everything works until I try to set up a Replica set. When I run rs.initiate() I get the "...replSetHeartbeat requires authentication... " error seen below. I can connect to the "hulk" server from "hawkeye" manually by passing --host "nodeserver-hulk:27017. Any Ideas?

system

Ubuntu Server 18.04.4 LTS
Mongod v.4.2.3

rs.initiate command

rs.initiate(
   {
      _id: "r1",
      version: 1,
      members: [
         { _id: 0, host : "nodeserver-hulk:27017"},
         { _id: 1, host : "nodeserver-hawkeye:27017"}
      ]
   }
)

rs.initiate error

{
        "ok" : 0,
        "errmsg" : "replSetInitiate quorum check failed because not all proposed set members responded affirmatively: nodeserver-hawkeye:27017 failed with command replSetHeartbeat requires authentication",
        "code" : 74,
        "codeName" : "NodeNotFound"
}

mongod.conf file(same for both servers)

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /mnt/mongo/data
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:
security:
  authorization: 'enabled'
#operationProfiling:

#replication:
replication:
   replSetName: 'r1'

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

show users command

> show users
{
        "_id" : "admin.james",
        "userId" : UUID("3ed97f2e-de49-4b98-84c8-566b34805863"),
        "user" : "james",
        "db" : "admin",
        "roles" : [
                {
                        "role" : "userAdminAnyDatabase",
                        "db" : "admin"
                },
                {
                        "role" : "dbOwner",
                        "db" : "admin"
                },
                {
                        "role" : "clusterAdmin",
                        "db" : "admin"
                },
                {
                        "role" : "readWriteAnyDatabase",
                        "db" : "admin"
                }
        ],
        "mechanisms" : [
                "SCRAM-SHA-1",
                "SCRAM-SHA-256"
        ]
}

Answer 1

You need to add a keyFile in your security config so that each node can authenticate against the others.

https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/

security:
  keyFile: <path-to-keyfile>