Connecting to MySQL database slow, takes n seconds

Question

Why is connecting to my InnoDB database often delayed by an integer amount of seconds?

Some background

I have a very small MySQL database, consisting of a table 'users' (150 records) and a table 'connections' (growing to 150*150 records). Tables and indexes add up to less than 5 MB.

When users are active, 5-50 records in 'connections' are changed (the weight is changed) or added (if they didn't exist yet). The whole app runs smoothly and load times are below ~100 ms.

Except when they are not.

The details

Under even quite small loads, page load times spike from 60 ms to somewhere between 1,000 ms and 10,000 ms.

Using the profiler in Symfony, I could pin down the delay for 95+% on the 'getRepository' statement, while the queries took only ~1 ms per query. This led me to believe that connecting to the database was the slow action. I wrote a helper script regularly connecting to the database to test this theory.

<?php // call this script commandline using watch
$a = microtime(true);
$pdo = new PDO('mysql:host=127.0.0.1;dbname=mydb','myuser','mypass');
file_put_contents( 'performance.txt', (microtime(true)-$a).PHP_EOL, FILE_APPEND );

The mystery

Connecting to the database took consistently 1-3 ms, or 1,001-1,003 ms, or 2,001-2,003 ms, or 3,001-3,003 ms, etc. An integer amount of seconds, plus the normal time. Nothing in between, like 400 ms or 800 ms. With no writes going on, the connection was made almost instantly. As soon as some writes were performed via the app, the higher numbers were reached.

What is causing this behavior? The InnoDB page_cleaner appears to do its work every 1,000 ms, maybe that's part of the explanation?

More importantly, how can I fix this? I was thinking of switching to MEMORY tables, but I'd say more elegant options should be available.

EDIT

On request, the variables and global status .

Additional information: I connect directly to 127.0.0.1 (see the code snippet above) and I tested the skip-name-resolve flag to no effect. It's a Debian server, by the way.

EDIT 2

I found the delays were either 1, 3, 7 or 15 seconds. Notice the pattern: 1 second, +2s, +4s, +8s. This really looks as some timeout issue...

Answer 1

It's common that reverse dns lookup takes a long time. Along with the size of the host_cache it can give a erratic behaviour.

Turn it off by adding this to my.cnf

[mysqld]
skip-name-resolve

Note that all grants must be by ip, not by name, if you change this.

There is more to read in the manual