Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon

Question

I am using MySQL Percona XtraDB Cluster with 3 node and MySQL service in one node did not work because of disk full. after fix the issue we try to start MySQL on this node but the following error occur: (before this issue all node had been working fine)

$ sudo /etc/init.d/mysql start

[....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
 failed!

$ sudo systemctl status mysql.service

● mysql.service - LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon
   Loaded: loaded (/etc/init.d/mysql; generated)
  Drop-In: /etc/systemd/system/mysql.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Tue 2020-03-24 11:57:55 +0430; 1h 2min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 19397 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

Mar 24 11:57:47 server-3 systemd[1]: Starting LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon...
Mar 24 11:57:47 server-3 mysql[19397]:  * Starting MySQL (Percona XtraDB Cluster) database server mysqld
Mar 24 11:57:55 server-3 mysql[19397]:  * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Mar 24 11:57:55 server-3 mysql[19397]:    ...fail!
Mar 24 11:57:55 server-3 systemd[1]: mysql.service: Control process exited, code=exited status=1
Mar 24 11:57:55 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 11:57:55 server-3 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.

Operating System: Ubuntu 18.04.2 LTS mysql Ver 14.14 Distrib 5.7.25-28, for debian-linux-gnu (x86_64) using 7.0

also please find journalctl log in below:

$ sudo journalctl -xe

Journal file /var/log/journal/7e072ef87bdf452f8a9684d905f87daf/user-1863206337.journal is truncated, ignoring file.
Mar 24 19:05:22 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:22.235+04:30] time="2020-03-24T19:05:22+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:22 server-3 mysql[6820]:  * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Mar 24 19:05:22 server-3 mysql[6820]:    ...fail!
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Control process exited, code=exited status=1
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mysql.service has failed.
--
-- The result is RESULT.
Mar 24 19:05:22 server-3 sudo[6779]: pam_unix(sudo:session): session closed for user root
Mar 24 19:05:22 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:22 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:22 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:22 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:22 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:23 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:54620->[192.168.8.3]:161
Mar 24 19:05:23 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:54647->[192.168.8.3]:161
Mar 24 19:05:25 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:25.550+04:30] time="2020-03-24T19:05:25+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:26 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:55485->[192.168.8.3]:161
Mar 24 19:05:26 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:55500->[192.168.8.3]:161
Mar 24 19:05:27 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:27.235+04:30] time="2020-03-24T19:05:27+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:27 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:27 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:27 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:27 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:27 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:30 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:30.550+04:30] time="2020-03-24T19:05:30+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:31 server-3 pmm-agent[30366]: ERRO[2020-03-24T19:05:31.438+04:30] cannot select @@slow_query_log_file: dial tcp 127.0.0.1:3306: connect: connection
Mar 24 19:05:32 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:32.234+04:30] time="2020-03-24T19:05:32+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:32 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:32 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:32 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:32 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:32 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:35 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:35.550+04:30] time="2020-03-24T19:05:35+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:36 server-3 sudo[7805]: admin_user : problem with defaults entries ; TTY=pts/0 ; PWD=/etc/systemd/system/mysql.service.d ; USER=root ;
Mar 24 19:05:36 server-3 sudo[7805]: admin_user : TTY=pts/0 ; PWD=/etc/systemd/system/mysql.service.d ; USER=root ; COMMAND=/bin/journalctl -xe
Mar 24 19:05:36 server-3 sudo[7805]: pam_unix(sudo:session): session opened for user root by admin_user(uid=0)

and please find mysqld.log in below:

/var/log$ sudo tail -n50 mysqld.log


2020-03-23T20:44:32.844540Z 36 [ERROR] WSREP: Node consistency compromised, aborting...
2020-03-23T20:44:32.844642Z 13 [ERROR] WSREP: Failed to apply trx: source: 89d92e92-6638-11e9-bdcb-7f2c26717188 version: 4 local: 0 state: APPLYING flags: 1 conn_id: 26113114 trx_id: 5580550846 seqnos (l: 151018291, g: 793430465, s: 793430462, d: 793430355, ts: 33700460567224654)
2020-03-23T20:44:32.844671Z 13 [ERROR] WSREP: Failed to apply trx 793430465 4 times
2020-03-23T20:44:32.844683Z 13 [ERROR] WSREP: Node consistency compromised, aborting...
2020-03-23T20:44:33.255602Z 21 [Note] WSREP: (c0cb4c75, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.8.1:4567 tcp://192.168.8.2:4567
2020-03-23T20:44:37.817120Z 21 [Note] WSREP: declaring node with index 0 suspected, timeout PT5S (evs.suspect_timeout)
2020-03-23T20:44:37.817139Z 21 [Note] WSREP: declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout)
2020-03-23T20:44:37.817147Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspecting node: 82cab7cb
2020-03-23T20:44:37.817152Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspected node without join message, declaring inactive
2020-03-23T20:44:37.817158Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspecting node: 89d92e92
2020-03-23T20:44:37.817161Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspected node without join message, declaring inactive
2020-03-23T20:44:37.817182Z 21 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,82cab7cb,31)
memb {
        c0cb4c75,0
        }
joined {
        }
left {
        }
partitioned {
        82cab7cb,0
        89d92e92,0
        }
)
2020-03-23T20:44:37.817203Z 21 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2020-03-23T20:44:37.817358Z 21 [Note] WSREP: gcomm: closed
2020-03-23T20:44:37.817463Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2020-03-23T20:44:37.817504Z 0 [Note] WSREP: Flow-control interval: [10000, 10000]
2020-03-23T20:44:37.817515Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-03-23T20:44:37.817522Z 0 [Note] WSREP: Received NON-PRIMARY.
2020-03-23T20:44:37.817524Z 13 [ERROR] WSREP: non-standard exception
2020-03-23T20:44:37.817530Z 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 793430465)
2020-03-23T20:44:37.817568Z 13 [Note] WSREP: applier thread exiting (code:8)
2020-03-23T20:44:37.817584Z 13 [Note] WSREP: Starting Shutdown
2020-03-23T20:44:37.817587Z 0 [Note] WSREP: Received self-leave message.
2020-03-23T20:44:37.817529Z 36 [ERROR] WSREP: non-standard exception
2020-03-23T20:44:37.817643Z 36 [Note] WSREP: applier thread exiting (code:8)
2020-03-23T20:44:37.817656Z 36 [Note] WSREP: Starting Shutdown
2020-03-23T20:44:37.817759Z 0 [Note] WSREP: Received shutdown signal. Will sleep for 10 secs before initiating shutdown. pxc_maint_mode switched to SHUTDOWN
2020-03-23T20:44:37.817813Z 0 [Note] WSREP: Flow-control interval: [10000, 10000]
2020-03-23T20:44:37.817841Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-03-23T20:44:37.817852Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2020-03-23T20:44:37.817860Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 793430465)
2020-03-23T20:44:37.817870Z 0 [Note] WSREP: RECV thread exiting 0: Success
2020-03-23T20:44:37.818185Z 21 [Note] WSREP: recv_thread() joined.
2020-03-23T20:44:37.818192Z 21 [Note] WSREP: Closing replication queue.
2020-03-23T20:44:37.818195Z 21 [Note] WSREP: Closing slave action queue.
2020-03-23T20:44:37.818247Z 21 [Note] WSREP: /usr/sbin/mysqld: Terminated.

How to start MySQL in this node?

thanks in advance.

Answer 1

As your myslqd.log shows, the disk full condition has left your node in a state, where it can find no way to achieve consistency with the cluster.

This implies, that you have to remove the node from the cluster, empty it out and then re-add it like a new node.