docker container wont start with Mysql docker image

Question

I use docker container with mysql offical images to create more than 11 database container, (container1 to container11). after setting up, all container running fine until container9. At container10, it only starts up about 1 mins and stop again. using docker logs to check container but I do not see anything. stop container9, and restart container10. It runs fine again. The situation seems only happen when I have 9 mysql container and trying to raise up 10th. If I stop one of them, and raise it up again. Then there is no problem. Is it bugs? or I miss some setting for docker bridge?

root@ec8dcb82f64d:/dev/shm# docker restart f4801b57c4cc
f4801b57c4cc
root@ec8dcb82f64d:/dev/shm# docker ps -a | grep f4801b57c4cc
f4801b57c4cc        mysql/mysql-server:5.7             "/entrypoint.sh my..."   2 weeks ago         Exited (1) 3 seconds ago                                                             db
root@ec8dcb82f64d:/dev/shm# docker logs f4801b57c4cc
Initializing database
Database initialized
MySQL init process in progress...
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.

/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*


MySQL init process done. Ready for start up.

root@ec8dcb82f64d:/dev/shm#

Answer 1

I think I hit the solution after one week even though I do not really understand what happens. the following is what I tried and so far, I can bring up mysql container up to 20 with no problems.

1: Try to create a dummpy mysql container for testing

$ docker run -e MYSQL_ROOT_PASSWORD=password mysql
Unable to find image 'mysql:latest' locally
latest: Pulling from library/mysql
...
Initializing database
2017-08-09T17:58:30.034595Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-08-09T17:58:30.039274Z 0 [Warning] InnoDB: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
2017-08-09T17:58:30.039294Z 0 [Warning] InnoDB: io_setup() attempt 1.
2017-08-09T17:58:30.539495Z 0 [Warning] InnoDB: io_setup() attempt 2.
2017-08-09T17:58:31.039701Z 0 [Warning] InnoDB: io_setup() attempt 3.
2017-08-09T17:58:31.539902Z 0 [Warning] InnoDB: io_setup() attempt 4.
2017-08-09T17:58:32.040115Z 0 [Warning] InnoDB: io_setup() attempt 5.
2017-08-09T17:58:32.540330Z 0 [ERROR] InnoDB: io_setup() failed with EAGAIN after 5 attempts.
2017-08-09T17:58:32.540378Z 0 [ERROR] InnoDB: Cannot initialize AIO sub-system
2017-08-09T17:58:32.540390Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2017-08-09T17:58:32.540401Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2017-08-09T17:58:32.540408Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-08-09T17:58:32.540412Z 0 [ERROR] Failed to initialize plugins.
2017-08-09T17:58:32.540415Z 0 [ERROR] Aborting

and hit the error code for io_setup() failed with EAGAIN

2: Examines the current value of aio-max-nr

$ sysctl fs.aio-max-nr
fs.aio-max-nr = 65536

3: Increase the value of aio-max-nr to 2097152

$ sudo sysctl -w fs.aio-max-nr=2097152

3: Start mysql service

4: Try to create more mysql-containers and bring up original one with no problems

Answer 2

Run docker events in background and then try starting your 10th container. You will surely see what's going wrong. Given below is an example to start an exited container and error while starting it. Commands below in sequence:

~$ sudo docker events &
[1] 9414

~$ sudo docker start 48137950f1b7
2017-08-03T00:01:18.971406558+05:30 network connect c79096ff0fef046d24b2a23907b3cc82c4df0838db2475909f8fa9f796a0418e (container=48137950f1b714797143529d63ec7221d3cbcd38bb6c8d20a241b06ddbd3d27a, name=bridge, type=bridge)
2017-08-03T00:01:19.305063392+05:30 container start 48137950f1b714797143529d63ec7221d3cbcd38bb6c8d20a241b06ddbd3d27a (image=ubuntu, name=modest_northcutt)
48137950f1b7
2017-08-03T00:01:19.305915636+05:30 container die 48137950f1b714797143529d63ec7221d3cbcd38bb6c8d20a241b06ddbd3d27a (exitCode=0, image=ubuntu, name=modest_northcutt)

Hope you will figure out.