Kubernetes: mysql pod failed to open log file /var/log/pods/

Question

I am following the official tutorial here to run a stateful mysql pod on the Kubernetes cluster which is already running on GCP. I have used the exact same commands to first create the persistent volume and persistent volume chain and then deployed the contents of the mysql yaml file as per the documentation. The mysql pod is not running and is in RunContainerError state. Checking the logs of this mysql pod shows:

failed to open log file "/var/log/pods/045cea87-6408-11e9-84d3-42010aa001c3/mysql/2.log": open /var/log/pods/045cea87-6408-11e9-84d3-42010aa001c3/mysql/2.log: no such file or directory

Update: As asked by @Matthew in the comments, the result of kubectl describe pods -l app=mysql is provided here:

Name:               mysql-fb75876c6-tk6ml
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-mycluster-default-pool-b1c1d316-xv4v/10.160.0.13
Start Time:         Tue, 23 Apr 2019 13:36:04 +0530
Labels:             app=mysql
                    pod-template-hash=963143272
Annotations:        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container mysql
Status:             Running
IP:                 10.52.0.7
Controlled By:      ReplicaSet/mysql-fb75876c6
Containers:
  mysql:
    Container ID:   docker://451ec5bf67f60269493b894004120b627d9a05f38e37cb50e9f283e58dbe6e56
    Image:          mysql:5.6
    Image ID:       docker-pullable://mysql@sha256:5ab881bc5abe2ac734d9fb53d76d984cc04031159152ab42edcabbd377cc0859
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       RunContainerError
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
      Exit Code:    128
      Started:      Tue, 23 Apr 2019 13:36:18 +0530
      Finished:     Tue, 23 Apr 2019 13:36:18 +0530
    Ready:          False
    Restart Count:  1
    Requests:
      cpu:  100m
    Environment:
      MYSQL_ROOT_PASSWORD:  password
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jpkzg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-jpkzg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jpkzg
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                                               Message
  ----     ------     ----              ----                                               -------
  Normal   Scheduled  32s               default-scheduler                                  Successfully assigned default/mysql-fb75876c6-tk6ml to gke-mycluster-default-pool-b1c1d316-xv4v
  Normal   Pulling    31s               kubelet, gke-mycluster-default-pool-b1c1d316-xv4v  pulling image "mysql:5.6"
  Normal   Pulled     22s               kubelet, gke-mycluster-default-pool-b1c1d316-xv4v  Successfully pulled image "mysql:5.6"
  Normal   Pulled     4s (x2 over 18s)  kubelet, gke-mycluster-default-pool-b1c1d316-xv4v  Container image "mysql:5.6" already present on machine
  Normal   Created    3s (x3 over 18s)  kubelet, gke-mycluster-default-pool-b1c1d316-xv4v  Created container
  Warning  Failed     3s (x3 over 18s)  kubelet, gke-mycluster-default-pool-b1c1d316-xv4v  Error: failed to start container "mysql": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system

As asked by @Hanx: Result of kubectl describe pv mysql-pv-volume

Name:            mysql-pv-volume
Labels:          type=local
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"mysql-pv-volume","namespace":""},"spec":{"a...
                 pv.kubernetes.io/bound-by-controller=yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    manual
Status:          Bound
Claim:           default/mysql-pv-claim
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        1Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/data
    HostPathType:
Events:            <none>

Result of kubectl describe pvc mysql-pv-claim

Name:          mysql-pv-claim
Namespace:     default
StorageClass:  manual
Status:        Bound
Volume:        mysql-pv-volume
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"accessModes":["R...
               pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
Events:        <none>

mysql-pv.yaml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

mysql.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

Answer 1

This is because you do not need to create those volumes and storageclasses on GKE. Those yaml files are completely valid if you would want to use minikube or kubeadm, but not in case of GKE which can take care of some of the manual steps on its own.

You can use this official guide to run mysql on GKE, or just use files edited by me and tested on GKE.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-volumeclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

And mysql Deployment:

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-volumeclaim

Make sure you read the linked guide as it explains the GKE specific topics there.