Sensu and Graphite. Configure transmission through AMQP

Question

I want to use Sensu as monitoring system and Graphite as backend for graphics.

I wish to configure Sensu for receiving data from RabbitMQ via AMQP protocol that's why I configured Carbon in such way:

# vim /etc/carbon/carbon.conf

# Enable AMQP if you want to receve metrics using an amqp broker
ENABLE_AMQP = True

# Verbose means a line will be logged for every metric received
# useful for testing
AMQP_VERBOSE = True

AMQP_HOST = 10.0.3.16
AMQP_PORT = 5672
AMQP_VHOST = /sensu
AMQP_USER = sensu
AMQP_PASSWORD = kubuntu710
AMQP_EXCHANGE = metrics_my
AMQP_METRIC_NAME_IN_BODY = True

Per my understanding Carbon with some frequency requests data from RabbitMQ (via AMQP) and save it via Whisper.

On other side Sensu saves some metrics in RabbitMQ, I configured it in next way:

root@sensu_server:/etc/sensu/conf.d# vim graphite_handler_amqp.json

{
  "handlers": {
    "graphite_amqp": {
      "type": "transport",
      "pipe": {
        "type": "topic",
        "name": "metrics_my",
        "durable": "true"
      },
      "mutator": "only_check_output"
    }
  }
}

And of course I attached this handler in such way:

root@sensu_server:/etc/sensu/conf.d# cat metrics_cpu.json
{
  "checks": {
    "metrics_cpu": {
      "type": "metric",
      "command": "/opt/sensu/embedded/bin/metrics-cpu-pcnt-usage.rb",
      "interval": 10,
      "subscribers": ["MONGO"],
      "handlers": ["graphite_amqp"]
    }
  }
}

Everything fine but Graphite can't draw metrics. This is log from Graphite side:

13/06/2016 18:57:16 :: New AMQP connection made

And this is from rabbitMQ on Sensu server side:

=INFO REPORT==== 13-Jun-2016::15:57:16 ===
accepting AMQP connection <0.25298.0> (10.0.3.95:43722 -> 10.0.3.16:5672)

=ERROR REPORT==== 13-Jun-2016::15:57:16 ===
Channel error on connection <0.25298.0> (10.0.3.95:43722 -> 10.0.3.16:5672, vhost: '/sensu', user: 'sensu'), channel 1:
operation exchange.declare caused a channel exception precondition_failed: "inequivalent arg 'durable' for exchange 'metrics_my' in vhost '/sensu': received 'true' but current is 'false'"

Why rabbitMQ thinks that "durable": set to "false", when Sensu should set it to true?

Can anybody share own expirience with such logic?

PS. Configuration with just tcp handler is working fine for me.

Answer 1

operation exchange.declare caused a channel exception precondition_failed: "inequivalent arg 'durable' for exchange 'metrics_my' in vhost '/sensu': received 'true' but current is 'false'"

The exchange metrics_my already exists and has the durable property set to false. Some other process is now trying to re-declare that same exchange with a different value for durable (true).

It looks like that when the processes start up they are trying to configure RabbitMQ using the configuration you have specified - making sure the required exchanges and queues exist.

However, RabbitMQ does not allow changing some properties of exchanges and queues after they have been created, so one of the processes is starting up, trying to make sure the exchange exists but is failing because it is specifying a different value for the durable property than what it already is.

My guess is that carbon and sensu have been configured to have a different value of durable for the metrics_my exchange.

Based on the snippets of configuration you provided, I don't see an option for changing the durable property for carbon, but you can for sensu.

You need to make everyone agree on what durable should be, delete the exchange (if durable will be different) and restart everything.

PS: The durable property specifies that the exchange should be persisted to disk and survive restarts of the RabbitMQ process.