Setting up a Sensu-Go cluster - cluster is not synchronizing

Question

I'm having an issue setting up my cluster according to the documents, as seen here: https://docs.sensu.io/sensu-go/5.5/guides/clustering/

This is a non-https setup to get my feet wet, I'm not concerned with that at the moment. I just want a running cluster to begin with.

I've set up sensu-backend on my three nodes, and have configured the backend configuration (backend.yml) accordingly on all three nodes through an ansible playbook. However, my cluster does not discover the other two nodes. It simply shows the following:

For backend1:

=== Etcd Cluster ID: 3b0efc7b379f89be
         ID                Name                Peer URLs              Client URLs       
 ────────────────── ─────────────────── ─────────────────────── ─────────────────────── 
  8927110dc66458af   backend1   http://127.0.0.1:2380   http://localhost:2379

For backend2 and backend3, it's the same, except it shows those individual nodes as the only nodes in their cluster.

I've tried both the configuration in the docs, as well as the configuration in this git issue: https://github.com/sensu/sensu-go/issues/1890

None of these have panned out for me. I've ensured all the ports are open, so that's not an issue.

When I do a manual sensuctl cluster member-add XX , I get an error message and it results in the sensu-backend process failing. I can't remove the member, either, because it causes the entire process to not be able to start. I have to revert to an earlier snapshot to fix it.

The configs on all machines are the same, except the IP's and names are appropriated for each machine

etcd-advertise-client-urls: "http://XX.XX.XX.20:2379"
etcd-listen-client-urls: "http://XX.XX.XX.20:2379"
etcd-listen-peer-urls: "http://0.0.0.0:2380"
etcd-initial-cluster: "backend1=http://XX.XX.XX.20:2380,backend2=http://XX.XX.XX.31:2380,backend3=http://XX.XX.XX.32:2380"
etcd-initial-advertise-peer-urls: "http://XX.XX.XX.20:2380"
etcd-initial-cluster-state: "new" # have also tried existing
etcd-initial-cluster-token: ""
etcd-name: "backend1"

Answer 1

Did you find the answer to your question? I saw that you posted over on the Sensu forums as well.

In any case, the easiest thing to do in this case would be to stop the cluster, blow out /var/lib/sensu/sensu-backend/etcd/ and reconfigure the cluster. As it stands, the behavior you're seeing seems like the cluster members were started individually first, which is what is potentially causing the issue and would be the reason for blowing the etcd dir away.