How to run etcd in multiple instance?

Question

I'm trying to run etcd with 3 instances in AWS, but I'm getting the error below when I try to list the members:

[ec2-user@etcd1 ~]$ etcdctl member list {"level":"warn","ts":"2021-10-19T04:50:01.981Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e0a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded

I'm starting my instances with the command below:

Instance 1:

etcd --data-dir=data.etcd --name etcd1
--initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380
--advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379
--initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
--initial-cluster-state new --initial-cluster-token token-01

Instance 2:

etcd --data-dir=data.etcd --name etcd2
--initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380
--advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379
--initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
--initial-cluster-state new --initial-cluster-token token-01

Instance 3:

etcd --data-dir=data.etcd --name etcd3
--initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380
--advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379
--initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
--initial-cluster-state new --initial-cluster-token token-01

Please, let me know if do you have some clue what I'm doing wrong.

Answer 1

For me it looks like you followed https://etcd.io/docs/v3.5/demo/#set-up-a-cluster and if this is the case, there are 2 things that you should change:

1. You used localhost in place of ${THIS_IP} variable when it should be vm IP address that is reachable from every vm in your cluster (it can be public or private address as long as it is reachable)
2. You used etcdX in --initial-cluster option and those should be IPs of vm's and not names of etcd members

What I'd recommend is setting those variables in every vm:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=etcd1
NAME_2=etcd2
NAME_3=etcd3
HOST_1=10.240.0.17 # <- should be ip of 1st vm
HOST_2=10.240.0.18 # <- should be ip of 2nd vm
HOST_3=10.240.0.19 # <- should be ip of 3rd vm
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

and then start etcd with below command for etcd1

THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
    --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

below command for etcd2

THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
    --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

and below for etcd3

THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
    --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Answer 2

Thank you wpedrak. I executed the commands how you mentioned, but now I'm getting other error:

{"level":"info","ts":"2021-10-19T11:38:23.882Z","caller":"etcdmain/etcd.go:72","msg":"Running: ","args":["etcd","--data-dir=data.etcd","--name","etcd1","--initial-advertise-peer-urls","http://13.58.134.34:2380","--listen-peer-urls","http://13.58.134.34:2380","--advertise-client-urls","http://13.58.134.34:2379","--listen-client-urls","http://13.58.134.34:2379","--initial-cluster","etcd1=http://13.58.134.34:2380,etcd2=http://18.118.5.45:2380,etcd3=http://18.119.98.88:2380","--initial-cluster-state","new","--initial-cluster-token","token-01"]} {"level":"info","ts":"2021-10-19T11:38:23.882Z","caller":"etcdmain/etcd.go:115","msg":"server has been already initialized","data-dir":"data.etcd","dir-type":"member"} {"level":"info","ts":"2021-10-19T11:38:23.882Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://13.58.134.34:2380"]} {"level":"info","ts":"2021-10-19T11:38:23.882Z","caller":"embed/etcd.go:367","msg":"closing etcd server","name":"etcd1","dat a-dir":"data.etcd","advertise-peer-urls":["http://13.58.134.34:2380"],"advertise-client-urls":["http://13.58.134.34:2379"]} {"level":"info","ts":"2021-10-19T11:38:23.883Z","caller":"embed/etcd.go:369","msg":"closed etcd server","name":"etcd1","data-dir":"data.etcd","advertise-peer-urls":["http://13.58.134.34:2380"],"advertise-client-urls":["http://13.58.134.34:2379"]} {"level":"fatal","ts":"2021-10-19T11:38:23.883Z","caller":"etcdmain/etcd.go:203","msg":"discovery failed","error":"listen tcp 13.58.134.34:2380: bind: cannot assign requested address","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\\n\\t/tmp/etcd-release-3.5.1/etcd/release/etcd/server/etcdmain/etcd.go:203\\ngo.etcd.io/etcd/server/v3/etcdmain.Main\\n\\t/tmp/etcd-release-3.5.1/etcd/release/etcd/server/etcdmain/main.go:40\\nmain.main\\n\\t/tmp/etcd-release-3.5.1/etcd/release/etcd/server/main.go:32\\nruntime.main\\n\\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}