MySQL 5.7 Percona server crashes and restarts frequently

Question

Since a couple of days my MySQL server is crashing and restart every other hours.

Things I've checked: Syslog is not giving me any indication

MySQL Error Log indicates:

    2018-08-22T10:28:25.602376Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 6702ms. The settings might not be optimal. (flushed=9, during the time.)
2018-08-22T10:34:21.112124Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4263ms. The settings might not be optimal. (flushed=8, during the time.)
Killed
2018-08-22T10:37:48.520909Z mysqld_safe Number of processes running now: 0
2018-08-22T10:37:48.522288Z mysqld_safe mysqld restarted
2018-08-22T10:37:48.595417Z 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 65535)
2018-08-22T10:37:48.595512Z 0 [Warning] Changed limits: max_connections: 214 (requested 500)
2018-08-22T10:37:48.595518Z 0 [Warning] Changed limits: table_open_cache: 400 (requested 10240)
2018-08-22T10:37:48.774434Z 0 [Warning] The use of InnoDB is mandatory since MySQL 5.7. The former options like '--innodb=0/1/OFF/ON' or '--skip-innodb' are ignored.
2018-08-22T10:37:48.774462Z 0 [Warning] The syntax 'avoid_temporal_upgrade' is deprecated and will be removed in a future release
2018-08-22T10:37:48.775643Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2018-08-22T10:37:48.778919Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.14-8-log) starting as process 27908 ...
2018-08-22T10:37:48.798784Z 0 [Warning] You need to use --log-bin to make --binlog-format work.
2018-08-22T10:37:48.807156Z 0 [Note] InnoDB: PUNCH HOLE support available
2018-08-22T10:37:48.807174Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-08-22T10:37:48.807180Z 0 [Note] InnoDB: Uses event mutexes
2018-08-22T10:37:48.807184Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-08-22T10:37:48.807189Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-08-22T10:37:48.807194Z 0 [Note] InnoDB: Using Linux native AIO
2018-08-22T10:37:48.808810Z 0 [Note] InnoDB: Number of pools: 1
2018-08-22T10:37:48.813108Z 0 [Note] InnoDB: Using CPU crc32 instructions
2018-08-22T10:37:48.815587Z 0 [Note] InnoDB: Initializing buffer pool, total size = 28G, instances = 8, chunk size = 128M
2018-08-22T10:37:49.681807Z 0 [Note] InnoDB: Completed initialization of buffer pool
2018-08-22T10:37:49.934256Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-08-22T10:37:49.943100Z 0 [Note] InnoDB: Recovering partial pages from the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2018-08-22T10:37:50.039106Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2018-08-22T10:37:50.250493Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 1253064250683
2018-08-22T10:37:50.569591Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 1253069493248

Tens of 'Doing Recovery' notices after.

Munin indicates to my that the InnoDB pool size could be the issue. And the logs have tons of these kind of messages:

2018-08-22T10:39:10.793549Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 7779ms. The settings might not be optimal. (flushed=11, during the time.)

my.cnf config settings

# INNODB #
innodb-flush-method            = O_DIRECT
innodb-log-files-in-group      = 2
innodb-log-file-size           = 1G
innodb-flush-log-at-trx-commit = 2
innodb-file-per-table          = 1
innodb-buffer-pool-size        = 28G

I'm kind of stuck on how to resolve the restarting.

Answer 1

Suggestions to consider for your my.cnf [mysqld] section (until you can post additional requested information)

innodb_fast_shutdown=0  # from 1 (YES) for clean (buffers flushed) to avoid recovery on start/restart
   yes shutdown will take a few seconds, maybe hours if
innodb_buffer_pool_pages_dirty are in the thousands or millions

innodb_lru_scan_depth=128  # from 1024 to conserve CPU every second  see refman
innodb_page_cleaners=4  # from 1 for additional page cleaning capacity
innodb_flushing_avg_loops=10  # from 30 to reduce the loop delay

also it is likely your Open Files limit is 1024. You can find out with ulimit -a from OS Command prompt. ulimit -n 20000 to raise the limit will help reduce file/table open frequency. Posting your additional requested information will provide more facts.