Kubernetes MongoDB pods with NFS persistent volume provisioning

Question

I have a kubernetes cluster and I have set up an NFS server as persistent volume for a mongodb deployment.

And I have set the PeristentVolume and PersistentVolumeClaim as below:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    name: mynfs 
spec:
  storageClassName: manual 
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <nfs-server-ip>
    path: "/srv/nfs/mydata" 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany 
  resources:
    requests:
      storage: 1Gi

Everything works fine but the only problem is, I can't run more than 1 mongodb pods because I get the following error.

{"t":{"$date":"2020-10-15T15:16:39.140+00:00"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"DBPathInUse: Unable to lock the lock file: /data/db/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /data/db directory"}}

That pod is always in CrashLoopBackOff and restarts and again to the same status.

I think the problem here is the same volume path mentioned in the mongodb deployment is trying to access by the two pods at the same time and when one pod is already have the lock, other pod failed.

Here's the mongodb deployment yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment    
  labels:
    name: mongodb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels: 
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        -  containerPort: 27017
        env: 
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-username
        - name: MONGO_INITDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-password
        volumeMounts:
        - name: data  
          mountPath: /data/db
      volumes: 
        - name: data
          persistentVolumeClaim: 
            claimName: task-pv-claim  

can someone please help me fix this?

Thank you.

Answer 1

Issue

This log entry already tells you what is the issue

{"t":{"$date":"2020-10-15T15:16:39.140+00:00"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"DBPathInUse: Unable to lock the lock file: /data/db/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /data/db directory"}}

All members access the same volume and data.

AFAIK you cannot have multiple instances of MongoDB pointing to the same path, each MongoDB instance needs to have exclusive access to its own data files.

Solution

You can run your application as StatefulSet with volumeClaimTemplate which ensures that each replica will mount its own volume. There is great answer about that.