GKE上的MongoDB Replicaset不起作用

Question

I'm creating 3 rc for each mongo pod:

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: mongo
  name: mongo-controller-1
spec:
  replicas: 1
  selector:
    name: mongo-1
  template:
    metadata:
      labels:
        name: mongo-1
        type: mongo
        pod: mongo-1
    spec:
      containers:
      - image: mongo
        command:
        - "/bin/bash"
        - "-c"
        - "service mongod stop; mkdir -p /srv/mongodb/db0 && mongod --dbpath /srv/mongodb/db0 --replSet rs0 && mongo --eval 'rs.initiate()'"
        name: mongo
        ports:
        - name: mongo
          containerPort: 27017
          hostPort: 27017
        volumeMounts:
            - name: mongo-persistent-storage-1
              mountPath: /data/db
            - name: mongo-js
              mountPath: /replicaset.js
      volumes:
        - name: mongo-persistent-storage-1
          gcePersistentDisk:
            pdName: mongo-1
            fsType: ext4
        - name: mongo-js
          hostPath:
            path: /home/sukhi/replicaset.js
      nodeSelector:
        role: mongo-1

---

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: mongo
  name: mongo-controller-2
spec:
  replicas: 1
  selector:
    name: mongo-2
  template:
    metadata:
      labels:
        name: mongo-2
        type: mongo
        pod: mongo-2
    spec:
      containers:
      - image: mongo
        command:
        - "/bin/bash"
        - "-c"
        - "/entrypoint.sh --port 27017 --smallfiles --replSet rs0"
        name: mongo
        ports:
        - name: mongo
          containerPort: 27017
          hostPort: 27017
        volumeMounts:
            - name: mongo-persistent-storage-2
              mountPath: /data/db
      volumes:
        - name: mongo-persistent-storage-2
          gcePersistentDisk:
            pdName: mongo-2
            fsType: ext4
      nodeSelector:
        role: mongo-2

---

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: mongo
  name: mongo-controller-3
spec:
  replicas: 1
  selector:
    name: mongo-3
  template:
    metadata:
      labels:
        name: mongo-3
        type: mongo
        pod: mongo-3
    spec:
      containers:
      - image: mongo
        command:
        - "/bin/bash"
        - "-c"
        - "/entrypoint.sh --port 27017 --smallfiles --replSet rs0"
        name: mongo
        ports:
        - name: mongo
          containerPort: 27017
          hostPort: 27017
        volumeMounts:
            - name: mongo-persistent-storage-3
              mountPath: /data/db
      volumes:
        - name: mongo-persistent-storage-3
          gcePersistentDisk:
            pdName: mongo-3
            fsType: ext4
      nodeSelector:
        role: mongo-3

And 4 services , 1 for each pod plus an overall service:

apiVersion: v1
kind: Service
metadata:
  labels:
    name: mongo
  name: mongo
spec:
  ports:
    - port: 27017
      targetPort: 27017
  selector:
    type: mongo

---

apiVersion: v1
kind: Service
metadata:
  labels:
    name: mongo-1
  name: mongo-1
spec:
  ports:
    - port: 27017
      targetPort: 27017
  selector:
    pod: mongo-1

---

apiVersion: v1
kind: Service
metadata:
  labels:
    name: mongo-2
  name: mongo-2
spec:
  ports:
    - port: 27017
      targetPort: 27017
  selector:
    pod: mongo-2

---

apiVersion: v1
kind: Service
metadata:
  labels:
    name: mongo-3
  name: mongo-3
spec:
  ports:
    - port: 27017
      targetPort: 27017
  selector:
    pod: mongo-3

After the pods are created I run:

kubectl exec mongo-controller-1asd -it -- mongo

to get a mongo shell and then run:

rs.initiate()

All's good so far and rs.conf() shows:

rs0:OTHER> rs.conf()
{
        "_id" : "rs0",
        "version" : 1,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 0,
                        "host" : "mongo-controller-1-ogwo7:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("57a458a59d13b31c826e27cd")
        }
}

The problem starts right after I add the first peer

rs0:PRIMARY> rs.add({host:"mongo-2.default.svc.cluster.local", priority:1})
{ "ok" : 1 } 

and check rs.status()

rs0:PRIMARY> rs.status()
2016-08-05T09:15:39.840+0000 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetGetStatus' on host '127.0.0.1:27017
'  :
DB.prototype.runCommand@src/mongo/shell/db.js:135:1
DB.prototype.adminCommand@src/mongo/shell/db.js:153:16
rs.status@src/mongo/shell/utils.js:1091:12
@(shell):1:1
2016-08-05T09:15:39.843+0000 I NETWORK  [thread1] trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2016-08-05T09:15:39.844+0000 I NETWORK  [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) ok
rs0:SECONDARY> rs.status()
{
        "set" : "rs0",
        "date" : ISODate("2016-08-05T09:15:46.289Z"),
        "myState" : 2,
        "term" : NumberLong(1),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongo-controller-1-ogwo7:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 207,
                        "optime" : {
                                "ts" : Timestamp(1470388513, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2016-08-05T09:15:13Z"),
                        "infoMessage" : "could not find member to sync from",
                        "configVersion" : 2,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "mongo-2.default.svc.cluster.local:27017",
                        "health" : 1,
                        "state" : 0,
                        "stateStr" : "STARTUP",
                        "uptime" : 23,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2016-08-05T09:15:43.295Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : -2
                }
        ],
        "ok" : 1
}

As you can see the PRIMARY I was on changes to SECONDARY and the new peer I just added gets stuck in STARTUP. I can't add any further nodes as I'm stuck without a primary and a non-functioning cluster.

Answer 1

I found a better way to add peers; once in the mongo shell create a config variable with the peers already defined, like so;

var config = {
    "_id" : "rs0",
    "members" : [
        {
            "_id" : 0,
            "host" : "mongo-1.default.svc.cluster.local"
        },
        {
            "_id" : 1,
            "host" : "mongo-2.default.svc.cluster.local"
        },
        {
            "_id" : 2,
            "host" : "mongo-3.default.svc.cluster.local",
        }
    ]
}

Then initialize the replicationset

rs.initiate(config)