Multiple MySQL Kubernetes pods with CrashLoopBackOff status

Question

I got an issue with the deployment of a WordPress + MySQL application on a Kubernetes cluster.

When using HorizontalPodAutoscaler to autoscale my wordpress and wordpress-mysql deployments, it works fine for the wordpress one but not the wordpress-mysql one.
Indeed, when multiple MySQL pods are created, some go in the CrashLoopBackOff status:

$ kubectl get all -n wordpress
NAME                                 READY     STATUS             RESTARTS   AGE
po/wordpress-3874566264-7031k        1/1       Running            0          16h
po/wordpress-mysql-898811424-2bdnn   0/1       CrashLoopBackOff   6          16h
po/wordpress-mysql-898811424-dxj92   1/1       Running            146        16h
po/wordpress-mysql-898811424-vs29j   0/1       CrashLoopBackOff   148        16h

NAME                  CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
svc/wordpress         10.254.121.190   10.0.0.13     80:30003/TCP   16h
svc/wordpress-mysql   None             <none>        3306/TCP       16h

NAME                  REFERENCE                    TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
hpa/wordpress         Deployment/wordpress         28% / 80%, 0% / 80%   1         10        1          16h
hpa/wordpress-mysql   Deployment/wordpress-mysql   90% / 80%, 0% / 80%   1         10        3          16h

NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/wordpress         1         1         1            1           16h
deploy/wordpress-mysql   3         3         3            1           16h

NAME                           DESIRED   CURRENT   READY     AGE
rs/wordpress-3874566264        1         1         1         16h
rs/wordpress-mysql-898811424   3         3         1         16h

And when I take a look at their logs, I get this:

$ kubectl logs -p wordpress-mysql-898811424-2bdnn -n wordpress
2018-09-12 08:04:12 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-09-12 08:04:12 0 [Note] mysqld (mysqld 5.6.41) starting as process 436 ...
2018-09-12 08:04:12 436 [Note] Plugin 'FEDERATED' is disabled.
2018-09-12 08:04:12 436 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-09-12 08:04:12 436 [Note] InnoDB: The InnoDB memory heap is disabled
2018-09-12 08:04:12 436 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-09-12 08:04:12 436 [Note] InnoDB: Memory barrier is not used
2018-09-12 08:04:12 436 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-09-12 08:04:12 436 [Note] InnoDB: Using Linux native AIO
2018-09-12 08:04:12 436 [Note] InnoDB: Using CPU crc32 instructions
2018-09-12 08:04:12 436 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-09-12 08:04:12 436 [Note] InnoDB: Completed initialization of buffer pool
2018-09-12 08:04:12 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:12 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:12 436 [Note] InnoDB: Retrying to lock the first data file
2018-09-12 08:04:13 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:13 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:14 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:14 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:15 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
[...]
2018-09-12 08:05:51 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:05:52 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [Note] InnoDB: Unable to open the first data file
2018-09-12 08:05:52 7f57a329f5c0  InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2018-09-12 08:05:52 436 [ERROR] InnoDB: Can't open './ibdata1'
2018-09-12 08:05:52 436 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' init function returned error.
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-09-12 08:05:52 436 [ERROR] Unknown/unsupported storage engine: InnoDB
2018-09-12 08:05:52 436 [ERROR] Aborting

2018-09-12 08:05:52 436 [Note] Binlog end
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'partition'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_METRICS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCKS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_TRX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'BLACKHOLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'ARCHIVE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MRG_MYISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MyISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MEMORY'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'CSV'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'sha256_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_old_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_native_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'binlog'
2018-09-12 08:05:52 436 [Note] mysqld: Shutdown complete

So it might be quite normal because each MySQL pod is trying to access ./ibdata1 at the same time, but then here is my question: is it really possible to have several MySQL pods sharing the same data? If the answer is yes, then how should I proceed to avoid these annoying errors?

If you need some other information, just tell me and I will edit my post.

Thank you in advance for your help!

Answer 1

So it might be quite normal because each MySQL pod is trying to access ./ibdata1 at the same time

Yes, if you try to do that (you didn't supply manifests), then that's the very reason you have CrashLoopBackOff state. First started instace will lock it and all subsequent will fail.

... trying to access ./ibdata1 at the same time ... is it really possible to have several MySQL pods sharing the same data?

If we talk about same data folder (the very same persistent volume, or hostpath or nfs share...) over two independent mysql instances - then no, not really, and not advisable for a number of reasons.

If you need to have multiple mysql instances (processes, containers or pods) sharing same data (not data folder!) you need to use replication (with read replicas or whatever...) where each of instances has own data folder structure but they sync data between them in some manner. Here is one example of a MySQL single-master topology with multiple slaves running asynchronous replication on kubernetes official documentation. Note that this is not a production setup, not HA setup, but just an illustration of simple replication scenario to give you an idea.

Now, some simple questions: are you sure you can't handle the load with single mysql instance that is serving several wordpress instances? Are you trying to make HA setup? Because answer to each of those question requires a bit different approach and architecture decisions than "increase number of pods from 1 up.

Answer 2

The port name is the same as any pod within your cluster. 
ex:

containers:
  - image: mysql:5.6
    name: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysql-pass
          key: password
    ports:
    - containerPort: 3306
      name: mysql-wiki
    volumeMounts:
    - name: mysql-persistent-storage
      mountPath: /var/lib/mysql
  volumes:
  - name: mysql-persistent-storage
    persistentVolumeClaim:
      claimName: mysql-pv-claim