Sensu Handler not triggering

Question

it seems my new Sensu Handler does not get called. But first, my config. in /etc/sensu/conf.d/checks.json:

{
"checks":
   "custom_check":{
      "command": "python3.6 /srv/custom_check.py",
      "subscribers": ["remote-checks"],
      "interval": 600,
      "ttl": 900,
      "handlers": ["custom_handler"],
      "source":"my-check-target"
   }
}

in /etc/sensu/conf.d/handlers.json:

{
  "handlers": {
    "custom_handler": {
      "type": "pipe",
      "command": "python3.6 /srv/custom-sensu-handlers/handler.py"
}

in the server logs, i see:

 {
  "timestamp":"2018-04-25T07:51:47.253449+0200",
  "level":"info",
  "message":"publishing check request",
  "payload":{"command":"python3.6 /srv/custom_checks/custom_check.py",
  "ttl":900,
  "handlers":["custom_handler"],
  "source":"my-check-target",
  "name":"custom_check",
  "issued":1524635507},
  "subscribers":["remote-checks"]
}

the client logs:

{
  "timestamp": "2018-04-30T06:24:00.012625+0200",
  "level": "info",
  "message": "received check request",
  "check": {
    "command": "python3.6 /srv/custom_checks/custom_check.py",
    "ttl": 900,
    "handlers": [
      "default",
      "custom_handler"
    ],
    "source": "my_check_target",
    "name": "custom_check",
    "issued": 1525062240
  }
}

{
  "timestamp": "2018-04-30T06:24:00.349912+0200",
  "level": "info",
  "message": "publishing check result",
  "payload": {
    "client": "assensu.internal.defaultoute.eu",
    "check": {
      "command": "python3.6 /srv/custom_checks/custom_check.py",
      "ttl": 900,
      "handlers": [
        "default",
        "custom_handler"
      ],
      "source": "my_check_target",
      "name": "custom_check",
      "issued": 1525062240,
      "subscribers": [
        "remote-checks"
      ],
      "interval": 600,
      "executed": 1525062240,
      "duration": 0.337,
      "output": "Check OK",
      "status": 0
    }
  }
}

And then, the logs stop to produce anything regarding the check. I can't find anything I am doing wrong. I even added a line of code to write to a logfile in the handler once it gets called, but nothing.
Any clues?
(if you are wondering, i am using python because i am not familiar with ruby...)

Answer 1

Handlers will only be executed on the following status types:

• warning
• critical
• unknown

https://docs.sensu.io/sensu-core/1.4/reference/handlers/#handler-attributes

Search for "severities", on how to customize this attribute in your handler definition.

An event is either create, resolve or flapping.

https://docs.sensu.io/sensu-core/1.2/reference/events/#event-actions

Your client log is showing "status: 0", which means the check passed, so no event is created and there's no reason to execute a handler for the event. Try setting your check to purposefully fail: sys.exit(2) , so that "status: 2" will be reported, and the handler will execute.

Handlers can deal with resolved event types. As an example, you might want to be notified via HipChat, Slack or Email that an event has cleared. (ie. gone from "status: 2", to "status: 0")

Python example of how we look at check event status:

if data['check']['status'] > 2:
    level = "Unknown"
elif data['check']['status'] == 2:
    level = "Critical"
elif data['check']['status'] == 1:
    level = "Warning"
elif data['check']['status'] == 0:
    level = "Cleared"

I would also make sure your /srv/custom-sensu-handlers/handler.py is owned by the sensu:sensu user/group recursively because it is outside of the default /etc/sensu/plugins directory.

https://docs.sensu.io/sensu-core/1.4/reference/handlers/#how-and-where-are-pipe-handler-commands-executed