MySql server deployed in kubernetes is very slow to respond

Question

I have deployed mysql in kubernetes. The pods are up and running. But when I tried to create db, table and insert data, all these operations seems to be very slow. Here is the yaml files I used for deployment. Can you look into the yaml and suggest me what could be the reason for it.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: "mysql"
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:8.0.20
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: mysql-persistent-storage
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: rbd-default
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  type: NodePort 
  ports:
  - port: 3306
    targetPort: 3306
  selector:
    app: mysql

I tried creating database after I exec into the pod, the operation took 40 sec to complete. When I tried connecting it to visual studio and perform same operation it took me more than 4 minutes. I think 40 sec itself is too long. However fetching data just took 300 ms from visual studio. I connected it to visual studio using IP and node port

Answer 1

Thank you all for spending time to answer the question. I think I solved the issue. It was basically the storage class that I used which was causing the issue. Once I updated it to rbd-fast, the response got much faster.

Answer 2

This sounds like a DNS lookup delay/timeout. If so, you need to set skip_name_resolve in your config file, in the [mysqld] section:

https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_skip_name_resolve

Note that you will not be able to use 'user'@'host' for access permissions if you do this, you will have to 'use user'@'ip_address' in your access permissions. You can still use the % wildcard, eg 'user'@'10.0.%' is valid. 'user'@'localhost' will also continue to work as this actually means local socket.