MySQL Database Performance Tuning

Question

I have developed a PHP and MySQL web application. Lately it has been showing signs of slowing down. I have ran mysql-tuner on 090516 with an uptime of 12 days.

The server specs are as follows:
- Linux CentOS 6
- 48 Processors
- 64GB RAM

Here is the mysql-tuner log output from 090516 with 12d uptime:

root@layer1 [~/mysqltuner]# perl mysqltuner.pl --outputfile /~/mysqltuner/result_mysqltuner.txt
String found where operator expected at mysqltuner.pl line 3096, near
        "get_wsrep_option 'gcache.mem_size'" (#1)
    (S syntax) The Perl lexer knows whether to expect a term or an operator.
    If it sees what it knows to be a term when it was expecting to see an
    operator, it gives you this warning.  Usually it indicates that an
    operator or delimiter was omitted, such as a semicolon.

        (Do you need to predeclare get_wsrep_option?)
Use of implicit split to @_ is deprecated at mysqltuner.pl line 3179 (#2)
    (D deprecated, W syntax) It makes a lot of work for the compiler when you
    clobber a subroutine's argument list, so it's better if you assign the results
    of a split() explicitly to an array (or list).

 >>  MySQLTuner 1.6.10 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.6.31-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 913M (Tables: 11)
[--] Data in InnoDB tables: 12M (Tables: 63)
[!!] Total fragmented tables: 11

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User 'munin@localhost' has user name as password.
[--] There are 605 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION
Argument "gcache.mem_size" isn't numeric in addition (+) at mysqltuner.pl line
        1965 (#3)
    (W numeric) The indicated string was fed as an argument to an operator
    that expected a numeric value instead.  If you're fortunate the message
    will identify which operator was so unfortunate.

Argument "gcache.mem_size" isn't numeric in addition (+) at mysqltuner.pl line
        1972 (#3)

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 12d 6h 21m 24s (2 q [0.000 qps], 1M conn, TX: 192B, RX: 240B)
[--] Reads / Writes: 100% / 0%
[--] Binary logging is disabled
[--] Physical Memory     : 62.8G
[--] Max MySQL memory    : 338.9M
[--] Other process memory: 505.0M
[--] Total buffers: 169.0M global + 1.1M per thread (151 max threads)
[--] P_S Max memory usage: 0B
Argument "*main::get_wsrep_option" isn't numeric in numeric ge (>=) at
        mysqltuner.pl line 285 (#3)
Argument "*main::get_wsrep_option" isn't numeric in numeric ge (>=) at
        mysqltuner.pl line 288 (#3)
Argument "*main::get_wsrep_option" isn't numeric in numeric ge (>=) at
        mysqltuner.pl line 291 (#3)
[--] Galera GCache Max memory usage: *main::get_wsrep_optionB
[OK] Maximum reached memory usage: 340.0M (0.53% of installed RAM)
[OK] Maximum possible memory usage: 338.9M (0.53% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/2)
[!!] Highest connection usage: 100%  (152/151)
[OK] Aborted connections: 0.00%  (34/1098331)
[OK] Query cache is disabled by default due to mutex contention.
[OK] No Sort requiring temporary tables
[OK] No joins without indexes
[OK] No tmp tables created on disk
[OK] Thread cache hit rate: 98% (17K created / 1M connections)
[OK] Table cache hit rate: 100% (381 open / 0 opened)
[OK] Open file limit used: 2% (228/10K)
[!!] Table locks acquired immediately: 94%

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B

-------- MyISAM Metrics ----------------------------------------------------------------------------
[OK] Key buffer used: 100.0% (8M used / 8M cache)
[!!] Key buffer size / total MyISAM indexes: 8.0M/147.7M
[!!] Read Key buffer hit rate: 87.6% (16M cached / 1M reads)
[!!] Write Key buffer hit rate: 56.8% (1M cached / 777K writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB buffer pool / data size: 128.0M/12.1M
[!!] InnoDB buffer pool <= 1G and innodb_buffer_pool_instances(!=1).
[!!] InnoDB Used buffer: 17.82% (1460 used/ 8192 total)
[OK] InnoDB Read buffer efficiency: 100.00% (290756388 hits/ 290757478 total)
[!!] InnoDB Write Log efficiency: 12.12% (3691 hits/ 30446 total)
[OK] InnoDB log waits: 0.00% (0 waits / 34137 writes)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] This is a standalone server.

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Set up a Secure Password for user@host ( SET PASSWORD FOR 'user'@'SpecificDNSorIp' = PASSWORD('secure_password'); )
    Reduce or eliminate persistent connections to reduce connection usage
    Optimize queries and/or use InnoDB to reduce lock wait
Variables to adjust:
    max_connections (> 151)
    wait_timeout (< 28800)
    interactive_timeout (< 28800)
    key_buffer_size (> 147.7M)
    innodb_buffer_pool_instances (=1)

I few things jump out at me, however I am not an experienced MySQL DBA and therefore am seeking advice from the stack community. Please advise which settings I should modify in my.cnf, and if I need to defragment tables in PHPMYADMIN, does this need to be done during off-hours.

I have not noticed my web app slowing down until just recently and recently I did reboot the server after a kernal update. Current uptime as of this post is 19hr.

**Extra information: ** The web app has a frontend where users login using mobile devices, and a backend where office employees manage the submissions (which are forms) and send them to different departments.

I can update this with munin charts if this is needed. I understand the server is grossly overprovisioned (meaning it has way more RAM and processors than necessary, but I wanted to have more than not enough down the road). My question is, can I just simply tweak some my.cnf settings (as suggested by mysqltuner and/or as suggested by phpmyadmin) to improve efficiency?

Answer 1

In the next ten years MySQL is unlikely to be refactored in a way that makes it use more than about four processors efficiently. So, your provisioning wastes 44 processor cores. That seems like too many. Also, it looks like MySQL is using less than 1GiB of your memory.

Ordinarily, MySQL is built out, not up. Meaning: most people who scale MySQL-based apps do so by using mirrors of the database. MariaDB's latest versions have a multimaster mirroring scheme.

The only obvious area where you have trouble is in connection churning. You should probably edit your my.cnf file to increase the number of simultaneous connections you use.

You need to investigate whether your web app uses connection pooling. It takes a while for a web app to establish a connection to the database, and if you're re-stablishing them for every web request that connection-opening operation can be a bottleneck. Your statistics show more than one new connection a second for 24 hours a day times twelve days. Unless your workload is very steadily spread out over all 24 hours in the day, that means your web app is peaking at a very high new-connection rate. This is likely to be the cause of your slowdown.

You didn't say what kind of web application platform you're using. Most platforms have connection pooling operations. Most web apps use a single set, or maybe two sets, of DBMS credentials (usernames) do do all the web app operations, so they can reuse the connections in the pool. Your stats say you have 600+ distinct users on the database. That is a lot for a web app.

Also (this is counterintuitive) sometimes too many web-app threads can harm performance. Sometimes it's better for user experience to restrict the number of threads, and let the web servers queue up the requests from clients, rather than trying to do too many operations at once. The TCP stacks on modern servers allow a reasonably deep queue of SYN requests, so the queuing won't be noticeable.

This is opinion based on the limited data you've shown: Your bottleneck isn't MySQL.