can't collect docker metrics using prometheus

Question

I want to monitor the Docker engine. I am following " Collect Docker metrics with Prometheus " to configure the Docker daemon for metrics which can be later collected and analyzed with Prometheus.

On my PC (OS version is CentOS 7.4, Docker version is 17.12.0.ce), I have pasted the following to /etc/docker/daemon.json

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Then, starting the Prometheus instance, I found the connection between Prometheus and Docker daemon is failed.

Error log

Get http://localhost:9323/metrics: dial tcp 127.0.0.1:9323: connect: connection refused

What can I do next?

Answer 1

Get docker ip addres

ip addr show docker0    

Enter docker ip addres in prometherus.yml configuration

    static_configs:
  - targets: ['172.17.0.1:9323']

Answer 2

In respect to your question

What can I do next?

after changing the configuration in /etc/docker/daemon.json it will be necessary to

systemctl daemon-reload
systemctl restart docker 

You may check after this if the port 9323 is in state LISTEN

lsof -Pi TCP -a -c dockerd 

As you mentioned in your comment that

curl http://$(hostname):9323/metrics  

is working properly on your local host, it indicates a problem with(in) your network.

As the Prometheus service is usually running somewhere else you may check if there is a connection from the remote machine to your Docker host. For this you could use something like

root@prometheusHost:/# nc -vz dockerHost 9323

It will give you a hint if the connection is refused, eg by a firewall.

Answer 3

Configure the Docker daemon

NOTE: 0.0.0.0 ie all IP addresses on the local machine

/etc/docker/daemon.json

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Open firewall port 9323 --permanently

$ sudo firewall-cmd --zone=public --permanent --add-port=9323/tcp
success

A few checks

Check, remember to list --permanent configuration, or it won't show

$ sudo firewall-cmd --zone=public --permanent --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client http https ssh
  ports: 9323/tcp                            <= OPEN
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

$ sudo lsof -Pi TCP -a -c dockerd
COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
dockerd-c 28157 root   14u  IPv6 30732963      0t0  TCP *:9323 (LISTEN)
...

Check connection to localhost

nc -vz localhost 9323
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to ::1:9323.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check connection to host IP eg 10.223.37.14

$ nc -vz 10.223.37.14 9323
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.223.37.14:9323.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check the metrics acccess using curl

curl http://localhost:9323/metrics

or using the IP from another machine

curl http://10.223.37.14:9323/metrics

You will get a connection refused if the port is not open

curl http://10.223.37.14:9323/metrics
curl: (7) Failed to connect to 10.223.37.14 port 9323: Connection refused

but once it's open, you will be able to see the metrics

$ curl http://10.223.37.14:9323/metrics
# HELP engine_daemon_container_actions_seconds The number of seconds it takes to process each container action
# TYPE engine_daemon_container_actions_seconds histogram
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.005"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.01"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.025"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.05"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.1"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.25"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="0.5"} 1
engine_daemon_container_actions_seconds_bucket{action="changes",le="1"} 1
...