进程在 ubuntu 18.04 上的 docker 容器中运行缓慢

Question

Processes are taking time when running them in a docker container on ubuntu 18 machine. But the same process with the same docker version is running fine on ubuntu 16 machine.

I have a node application listening on some port. Accepting get requests on the path "/" and "/docker" which simply runs a command "whoami" in the host machine and in docker container respectively and returns the result. The same node application with the same docker container is running on both the machines (ubuntu16 and ubuntu18).

Now, First I tried sending 20 concurrent get request with path "/" to both the machines. And both the machines executed the command in avg of
35-40ms.

Now, Second I tried sending 20 concurrent get request with path "/docker" to both the machines. Here, ubuntu16 machine took Max 4.3 seconds and Avg 3 seconds. But ubuntu18 machine taking Max 10seconds and Avg 9 seconds.

I tried the above test multiple times. and concluded when running process inside docker the time taken to execute is almost double in ubuntu18 machine as compare to ubuntu16.

I checked the following:

1. I tried monitoring through top and htop while hitting 20 requests. But everything seems the same their.

2. Also tried monitoring using perf command. But unable to find any unusual difference their. But I am not very used to perf command and so unable to understand clearly.

3. While these 20 requests were in processing. I run same docker command manually with strace . And found random results ie sometimes time taken in clock_gettime or futex (FUTEX_WAIT) or sometimes in +++ exited with 0 +++ message on ubuntu18. and but less time took on ubuntu16.

Below are my different configurations and code snippet I am using and running.

node    v10.16.0
npm     6.9.0
docker  18.09.8
ubuntu  16.04.3 LTS, xenial


node    v10.16.0
npm     6.9.0
docker  18.09.8
ubuntu  18.04.2 LTS, bionic

Node application code snippet:

// for path "/docker"
var excuteInDocker = function() {
    var cmd = "docker";
    var args = ["exec", "ubuntu", "whoami"];
    return executeCmd(cmd, args);
}

// for path "/"
var execute = function(){
    var cmd = 'whoami';
    var args = [];
    return executeCmd(cmd, args);
}

Ouput of docker info which are common to both ubuntu 16 and 18:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.09.8
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.296GiB
Name: myhostname
ID: LLLO:OMTS:PNNM:T3MP:AD2F:UMDG:IIZK:OGBO:3ZLL:YDBX:ONAO:AY5G
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 27
 Goroutines: 42
 System Time: 2019-07-25T15:25:54.991694211+05:30
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

docker info specific to ubuntu 16

Kernel Version: 4.4.0-112-generic
Operating System: Ubuntu 16.04.3 LTS
Total Memory: 7.303GiB
ID: FOFI:RW7N:RZSP:HHKH:BKS3:LMWL:TC2J:W7V2:222Y:Q2AU:XMU3:KLU7

docker info specific to ubuntu 18

Kernel Version: 4.15.0-1040-aws
Operating System: Ubuntu 18.04.2 LTS
Total Memory: 7.296GiB
ID: LLLO:OMTS:PNNM:T3MP:AD2F:UMDG:IIZK:OGBO:3ZLL:YDBX:ONAO:AY5G

---

ubuntu16 machine Data:

1. Data of time taken in execution

2019-07-25 14:06:42.851 INFO  uid: 540ae880-aeb7-11e9-919d-dd32b3cf84d5 time: 475 result: {"success":true,"data":"root"}
2019-07-25 14:06:43.183 INFO  uid: 54145e60-aeb7-11e9-919d-dd32b3cf84d5 time: 745 result: {"success":true,"data":"root"}
2019-07-25 14:06:45.711 INFO  uid: 540c4810-aeb7-11e9-919d-dd32b3cf84d5 time: 3326 result: {"success":true,"data":"root"}
.
.
.
2019-07-25 14:06:46.835 INFO  uid: 541d5f10-aeb7-11e9-919d-dd32b3cf84d5 time: 4338 result: {"success":true,"data":"root"}

2. logs of command strace -t docker exec ubuntu whoami enter image description here

3. result of perf top --sort comm,dso enter image description here

---

ubuntu18 machine Data:

1. Data of time taken in execution

2019-07-25 14:07:32.559 INFO  uid: 715a6af0-aeb7-11e9-a5a9-2fffd4e800d1 time: 1008 result: {"success":true,"data":"root"}
2019-07-25 14:07:32.941 INFO  uid: 7178c860-aeb7-11e9-a5a9-2fffd4e800d1 time: 1191 result: {"success":true,"data":"root"}
2019-07-25 14:07:40.363 INFO  uid: 71767e70-aeb7-11e9-a5a9-2fffd4e800d1 time: 8628 result: {"success":true,"data":"root"}
.
.
.
2019-07-25 14:07:41.970 INFO  uid: 718af0d0-aeb7-11e9-a5a9-2fffd4e800d1 time: 10101 result: {"success":true,"data":"root"}

2. logs of command strace -t docker exec ubuntu whoami enter image description here

3. result of perf top --sort comm,dso enter image description here

So, I need help in debugging what is wrong with docker on the ubuntu18 machine. Or if there is any limitation with docker on ubuntu18 or maybe some machine or docker configuration issue.

Answer 1

I did not encounter such a problem on my desktop, but mysql container was running very slowly in a way that I could not understand on my ubuntu laptop. This solution solved my problem.