kubernetes mysql statefulset not taking new password, though password changes in env

Question

I have created statefulset of mysql using below yaml with this command:

• kubectl apply -f mysql-statefulset.yaml

Yaml:

apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  labels:
    app: mysql
spec:
  ports:
  - port: 3306
    name: db
  clusterIP: None
  selector:
    app: mysql
---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-sts
spec:
  selector:
    matchLabels:
      app: mysql # has to match .spec.template.metadata.labels
  serviceName: mysql-service
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: mysql # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mysql
        image: mysql
        env:
          - name: MYSQL_ROOT_PASSWORD
            value: "okaoka"
        ports:
        - containerPort: 3306
          name: db
        volumeMounts:
        - name: db-volume
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: db-volume
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: standard
      resources:
        requests:
          storage: 1Gi

After that 3 pods and for each of them a pvc and pv was created. I successfully entered one of the pod using:

• kubectl exec -it mysql-sts-0 sh

and then login in mysql using:

• mysql -u root -p

after giving this command a:

• Enter password:

came and I entered the password:

• okaoka

and successfully could login. After that I exited from the pod.

Then I deleted the statefulset (as expected the pvc and pv were there even after the deletion of statefulset). After that I have applied a new yaml slightly changing the previous one, I changed the password in yaml, gave new password:

• okaoka1234

and rest of the yaml were same as before. The yaml is given below, now after applying this yaml (only changed the password) by:

• kubectl apply -f mysql-statefulset.yaml

it successfully created statefulset and 3 new pods (who binded with previous pvc and pv, as expected).

Changed Yaml:

apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  labels:
    app: mysql
spec:
  ports:
  - port: 3306
    name: db
  clusterIP: None
  selector:
    app: mysql
---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-sts
spec:
  selector:
    matchLabels:
      app: mysql # has to match .spec.template.metadata.labels
  serviceName: mysql-service
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: mysql # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mysql
        image: mysql
        env:
          - name: MYSQL_ROOT_PASSWORD
            value: "okaoka1234"   # here is the change
        ports:
        - containerPort: 3306
          name: db
        volumeMounts:
        - name: db-volume
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: db-volume
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: standard
      resources:
        requests:
          storage: 1Gi

Now the problem is when I again entered a pod using:

• kubectl exec -it mysql-sts-0 sh

then used:

• mysql -u root -p

and again the:

• Enter password:

came and this time when I gave my new password:

• okaoka1234

it gave access denied.

When I printed the env (inside the pod) using:

• printenv

then I could see that:

• MYSQL_ROOT_PASSWORD=okaoka1234

that means in environment variable it changed and took the new password, but I could not logged in by the new password.

The interesting thing is that I could logged in by giving my previous password okaoka , I don't know why it is taking the previous password in this scenario not the new one which is even in the env (inside pod) also. Can anybody provide the logic behind this?

Answer 1

Most probably, the image that you are using in your StatefulSet, uses the environment variable as a way to initialize the password when it creates for the first time the structure on the persisted storage (on its pvc).

Given the fact that the pvc and pv are the same of the previous installation, that step is skipped, the database password is not updated, since the database structure is already found in the existing pvc.

After all, the root user is just a user of the database, its password is stored in the database. Unless the image applies any particular functionality at its start with its entrypoint, it makes sense to me that the password remain the same.

What image are you using? The docker hub mysql image or a custom one?

Update

Given the fact that you are using the mysql image on docker hub, let me quote a piece of the entrypoint ( https://github.com/docker-library/mysql/blob/master/template/docker-entrypoint.sh )

    # there's no database, so it needs to be initialized
    if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
        docker_verify_minimum_env

        # check dir permissions to reduce likelihood of half-initialized database
        ls /docker-entrypoint-initdb.d/ > /dev/null

        docker_init_database_dir "$@"

        mysql_note "Starting temporary server"
        docker_temp_server_start "$@"
        mysql_note "Temporary server started."

        docker_setup_db
        docker_process_init_files /docker-entrypoint-initdb.d/*

        mysql_expire_root_user

        mysql_note "Stopping temporary server"
        docker_temp_server_stop
        mysql_note "Temporary server stopped"

        echo
        mysql_note "MySQL init process done. Ready for start up."
        echo
    fi

When the container starts, it makes some checks and if no database is found (and the database is expected to be on the path where the persisted pvc is mounted) a series of operations are performed, creating it, creating default users and so on.

Only in this case, the root user is created with the password specified in the environment (inside the function docker_setup_db)

Should a database already be available in the persisted path, which is your case since you let it mount the previous pvc, there's no initialization of the database, it already exists.

Everything in Kubernetes is working as expected, this is just the behaviour of the database and of the mysql image. The environment variable is used only for initialization, from what I can see in the entrypoint.

It is left to the root user to manually change the password, if desired, by using a mysql client.