Error Setting up Containerized Sensu-server and client, to monitor Docker

Question

I am trying to setup sensu-server to monitor my docker containers, by following the instructions on this post.

The Docker daemon is running on the same host has the sensu-server and sensu-clients (localhost).

The file check-docker, contains an entry to call the script load-docker-metrics.sh

check-docker.sh

{
  "checks": {
    "load_docker_metrics": {
      "type": "metric",
      "command": "load-docker-metrics.sh",
      "subscribers": [
        "docker"
      ],
      "interval": 10
    }
  }
}

load-docker-metrics.sh

#!/bin/bash
set -e

# Count all running containers
running_containers=$(echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock \
    | tail -n +5                                                           \
    | python -m json.tool                                                  \
    | grep \"Id\"                                                          \
    | wc -l)
# Count all containers
total_containers=$(echo -e "GET /containers/json?all=1 HTTP/1.0\r\n" | nc -U /var/run/docker.sock \
 | tail -n +5 \
 | python -m json.tool \
 | grep \"Id\" \
 | wc -l)

# Count all images
total_images=$(echo -e "GET /images/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock \
 | tail -n +5 \
 | python -m json.tool \
 | grep \"Id\" \
 | wc -l)

echo "docker.invismac.running_containers ${running_containers}"
echo "docker.invismac.total_containers ${total_containers}"
echo "docker.invismac.total_images ${total_images}"

if [ ${running_containers} -lt 3 ]; then
    exit 1;
fi

load-docker-metrics is basically making some calls to the docker api, and retrieving a few metrics. If I run this script on my host (invismac), it gives me an accurate result:

docker.invismac.running_containers 7
docker.invismac.total_containers 8
docker.invismac.total_images 15

I am launching the sensu-server container, with this command:

docker run -d --name sensu-server \\ -p 3000:3000 \\ -p 4567:4567 \\ -p 5671:5671 \\ -p 15672:15672 \\ -v $PWD/check-docker.json:/etc/sensu/conf.d/check-docker.json \\ hiroakis/docker-sensu-server

After this, I can correctly access the uchima dashboard, at port 3000

Then I launch the client, with this command:

docker run --name sensu-client --privileged \\ -v $PWD/load-docker-metrics.sh:/etc/sensu/plugins/load-docker-metrics.sh \\ -v /var/run/docker.sock:/var/run/docker.sock \\ usman/sensu-client localhost sensu password CLIENT-1 localhost

The client is suppose to communicate with RabbitMQ, and register itself in uchima, but nothing happens. I have entered the container that runs the client, and by looking at the log, there seems to be an error with rabbitMQ:

"timestamp":"2016-04-15T15:18:50.768594+0000","level":"error","message":"[amqp] Detected TCP connection failure"}

Rabbitmq is running, and I can access to its endpoint here:

http://localhost:15672

The dashboard gives me some information: I have 2 connections, 2 channels, 2 users.

Also, if I go to the sensu-master container, I can check that rabbitmq has this user:

sensu [administrator]

With all permissions:

/sensu .* .* .*

Any ideas why the sensu-client is not able to communicate with RabbitMQ? Some other check that I could try? Am I passing some wrong values on the configuration of the server or the client? is it not possible to have server and client in the same host?

Dmesg on the client, also gives me this message (I don't know if it could be related)

[29252.322147] audit: type=1400 audit(1460734746.650:975): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=27531 comm="ps" requested_mask="trace" denied_mask="trace" peer="unconfined"

Answer 1

I have the same problem. Did everything by this instruction, looks very similar like yours. The only way I solved the problem was to start the server and client parts on different machines with different IP's.

After this you will have 3 connections in RabitMQ, no 'ampq' errors in sensu-client log, and successfully added client in Uchiwa.

Here how it looks like.