无法启动 MySQL 社区服务器 -> InnoDB 强制恢复错误 -> InnoDB mmamp 错误

Question

I am running a AWS Lightsail instance with Ubuntu 18.04 and latest MySQL. Since a few days, the MySQL service stops at random times and then forces me to perform a manual restart after which everything goes back to normal for 24-48 hours until I face another stop.

Terminal echoes this error when running sudo service mysql status :

    ● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2020-04-26 17:54:44 UTC; 10h ago
  Process: 30117 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
  Process: 30095 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 14182 (code=killed, signal=KILL)

Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Failed to start MySQL Community Server.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 8.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Stopped MySQL Community Server.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Start request repeated too quickly.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Failed to start MySQL Community Server.

After digging through the /var/log/mysql/error.log file, I found dozens of lines like this:

2020-04-26T10:10:46.710146Z 2 [ERROR] InnoDB: innodb_force_recovery is on. We do not allow database modifications by the user. Shut down mysqld and edit my.cnf to$

Since the error message is truncated at exactly the wrong spot, I did some more digging through other Warning messages and found this one:

2020-04-26T17:54:44.312663Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2020-04-26T17:54:44.312671Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2020-04-26T17:54:44.312675Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-04-26T17:54:44.312680Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-04-26T17:54:44.312684Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-04-26T17:54:44.312688Z 0 [ERROR] Failed to initialize builtin plugins.
2020-04-26T17:54:44.312691Z 0 [ERROR] Aborting

I am by no means a MySQL expert, but I interpret that mmamp error as the system running out of space, which is a little weird since the instance uses around 10% space and has at most 2 users at the same time (hence little to no CPU usage). I run a single site Wordpress installation (just to divert users) and phpmyadmin plus some minor testing scripts.

Appreciate any help with this, thank you.

Answer 1

The most likely cause is that your configuration is broken in a way that makes mysqld try to allocate more memory that your server has.