mysql container won't start on kubernetes when backed by NFS Dynamic provisioner

Question

I'm having issues getting the mysql container starting properly. But to sum it up, with the nfs dynamic provisioner the mysql container won't start and throws an error of mkdir: cannot create directory '/var/lib/mysql/': File exists even though the NFS mount is in the container, and appears to be functioning properly.

I installed Dyanamic NFS provisioner installed on my K8 cluster from here https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client . The test claim and test pod they show on the instructions work.

Now to run mysql, I took the code snippets from here:

https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/

kubectl apply mysql-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: managed-nfs-storage  <--- THIS MATCHES MY NFS STORAGECLASS
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

kubectl apply -f mysql-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS          REASON   AGE
persistentvolume/mysql-pv-volume                            20Gi       RWO            Retain           Bound    default/mysql-pv-claim         managed-nfs-storage            5m16s

NAMESPACE      NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
default        persistentvolumeclaim/mysql-pv-claim    Bound    mysql-pv-volume                            20Gi       RWO            managed-nfs-storage   5m27s

The pv was created automatically by the dynamic provisioner

Get the error...

$ kubectl logs mysql-7d7fdd478f-l2m8h
2020-03-05 18:26:21+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.47-1debian9 started.
2020-03-05 18:26:21+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-03-05 18:26:21+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.47-1debian9 started.
mkdir: cannot create directory '/var/lib/mysql/': File exists

This error stops the container from starting...

I went and deleted the deployment and added command: [ "/bin/sh", "-c", "sleep 100000" ] so the container would start...

After getting into the container, I checked the NFS mount is properly mounted and is writable...

# df -h | grep mysql
nfs1.example.com:/k8/default-mysql-pv-claim-pvc-0808d1bd-69ca-4ff5-825a-b846b1133e3a  1.0T  1.6G 1023G   1% /var/lib/mysql

If I create a "local" pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

and created the mysql deployment, the mysql pod starts up without issue.

So at this point, with dynamic provisioning (potentially just on NFS?) the mysql container doesn't work.

Anyone have any suggestions?

Answer 1

I'm not exactly sure what is the cause of this so here is few options.

First you could try setting securityContext , because volume might be mounted without proper permissions.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false

You can find out the proper group id and user by typing id and gid inside the container. Or just using kubectl exec -it <pod-name> bash .

Second, try using subPath

volumeMounts:
- name: mysql-persistent-storage
  mountPath: "/var/lib/mysql"
  subPath: mysql

If that won't work I would test the NFS on another pod with initContainer that is creating a directory. And I would redo the whole nfs maybe using this guide .