sensu-client is failing during start

Question

My sensu-client is failing during start (fresh install) and /var/log/sensu/sensu-client.log doesn't show much despite me adding LOG_LEVEL=debug into /etc/default/sensu . I used similar client.json and rabbitmq.json config files (inside /etc/sensu/conf.d) on my other sensu-clients (copied ssl certificates).

   $ sudo service sensu-client start
    [FAILED] sensu-client[  OK  ]

Below is sensu-client log

$ tail -f /var/log/sensu/sensu-client.log 
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/daemon.rb:187:in `setup_transport'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/client/process.rb:412:in `start'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/client/process.rb:19:in `block in run'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `call'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/client/process.rb:18:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/bin/sensu-client:10:in `<top (required)>'
        from /opt/sensu/bin/sensu-client:23:in `load'
        from /opt/sensu/bin/sensu-client:23:in `<main>'
^C

Here is default config

$ cat /etc/default/sensu 
EMBEDDED_RUBY=false
LOG_LEVEL=debug

Even reboot of my RHEL7 doesnt help, see log below

Mar 25 13:09:24 nms02w sensu-client: Starting sensu-client[  OK  ]#015[FAILED]
Mar 25 13:09:24 nms02w systemd: sensu-client.service: control process exited, code=exited status=1
Mar 25 13:09:24 nms02w systemd: Unit sensu-client.service entered failed state.
Mar 25 13:09:24 nms02w systemd: sensu-client.service failed.

Adding more log:

 systemctl status sensu-client.service
● sensu-client.service - LSB: Sensu monitoring framework client
   Loaded: loaded (/etc/rc.d/init.d/sensu-client)
   Active: failed (Result: exit-code) since Fri 2016-03-25 13:09:24 EDT; 1h 47min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 948 ExecStart=/etc/rc.d/init.d/sensu-client start (code=exited, status=1/FAILURE)

Mar 25 13:09:21 nms02w systemd[1]: Starting LSB: Sensu monitoring framework client...
Mar 25 13:09:21 nms02w runuser[983]: pam_unix(runuser:session): session opened for user sensu by (uid=0)
Mar 25 13:09:24 nms02w sensu-client[948]: [38B blob data]
Mar 25 13:09:24 nms02w systemd[1]: sensu-client.service: control process exited, code=exited status=1
Mar 25 13:09:24 nms02w systemd[1]: Failed to start LSB: Sensu monitoring framework client.
Mar 25 13:09:24 nms02w systemd[1]: Unit sensu-client.service entered failed state.
Mar 25 13:09:24 nms02w systemd[1]: sensu-client.service failed.

Answer 1

I had the same problem with old version of sensu: 0.20.3-1 Full error message described here:

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:859:in `open_udp_socket': no datagram socket (RuntimeError)
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:859:in `open_datagram_socket'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/client/process.rb:355:in `setup_sockets'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/client/process.rb:413:in `start'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/client/process.rb:19:in `block in run'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `call'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/lib/sensu/client/process.rb:18:in `run'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-0.20.3/bin/sensu-client:10:in `<top (required)>'
from /opt/sensu/bin/sensu-client:23:in `load'
from /opt/sensu/bin/sensu-client:23:in `<main>'

The problem is `open_udp_socket': no datagram socket (RuntimeError) that means udp socket already in use, and sensu-client cannot bind it. Okey, you always can try to find halted process that used the socket:

root@xxx- /etc/sensu/conf.d/checks $ netstat -ntulp|grep 3030 
udp     0      0 127.0.0.1:3030   0.0.0.0:*                           603/sudo
root@xxx- /etc/sensu/conf.d/checks $ kill -9 603

After it sensu-client will start correctly.

root@xxx- /etc/sensu/conf.d/checks $ /etc/init.d/sensu-client start
 * Starting sensu-client                                [ OK ]

Kind regards