The connection to the server 10.117.xxx.xxx:6443 was refused - did you specify the right host or port?

Question

I have been trying to get a basic Kubernetes cluster running according to the following tutorial https://kubernetes.io/docs/setup/independent/install-kubeadm/

I started with an up-to-date ubuntu 16.04 system and installed docker.

wget -qO- https://get.docker.com/ | sed 's/docker-ce/docker-ce=18.06.3~ce~3-0~ubuntu/' | sh

After that I installed the kubelet / Kubeadm and kubectl modules

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

Made sure that swap etc was off sudo swapoff -a

Performed the init using sudo kubeadm init

[init] Using Kubernetes version: v1.13.3
...
To start using your cluster
...
mkdir ...
You can now join any ...
...

I make the .kube folder and config

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl cluster-info then shows

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 10.117.xxx.xxx:6443 was refused - did you specify the right host or port?

After giving it a few attempt I once received:

sudo kubectl cluster-info
Kubernetes master is running at https://10.117.xxx.xxx:6443
KubeDNS is running at https://10.117.xxx.xxx:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

But a second later its back to the permission denied

sudo kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 10.117.xxx.xxx:6443 was refused - did you specify the right host or port?

I tried with and without sudo... and sudo kubectl get nodes also refused to work.

The connection to the server 10.117.xxx.xxx:6443 was refused - did you specify the right host or port?

What am I missing that it won't connect?

ping 10.117.xxx.xxx works fine and even ssh to this address works and is the same server.

Edit

sudo systemctl restart kubelet.service shows that the cluster comes online but for some reason goes offline within minutes.

kubectl cluster-info
Kubernetes master is running at https://10.117.0.47:6443
KubeDNS is running at https://10.117.0.47:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
...
kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 10.117.0.47:6443 was refused - did you specify the right host or port?

edit2

After doing a full reset and using the following init...

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 

Followed by

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

Allowed me to install the pod network add-on but was only short-lived.

clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

kubectl cluster-info
Kubernetes master is running at https://10.117.xxx.xxx:6443
KubeDNS is running at https://10.117.xxx.xxx:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

➜  ~ kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 10.117.xxx.xxx:6443 was refused - did you specify the right host or port?

edit3

Removing all docker images, containers etc... and then perform a restart using sudo systemctl restart kubelet.service seems to do the trick for a few minutes but then all docker containers are killed and removed without any signal. How can I look into the logging of these containers to perhaps find out why they are killed?

https://pastebin.com/wWSWXM31 log file

Answer 1

I don't know the specifics of "Kubernetes", but i can explain what you are seeing.

The connection refused is not a permission denied. It means: "I contacted the server at that IP address and port and no server was running on that port."

So.... you will have to go to the remote system (the one that you keep calling 10.117.xxx.xxx) and doublecheck that the server is running. And on what port.

For example, the "netstat -a" tool will list all open ports and connections. You should see listening servers as

tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN     

here in my case it is listening on port 9090. You are looking for an entry with 6443. It probably won't be there, because that's what "connection refused" is already telling you. You need to start the server process that's supposed to provide that service and watch for errors. Check for errors in /var/log/syslog if you don't see them on your terminal.

Answer 2

After reboot got the same error. swapoff -a works for me

Answer 3

It's possible the kube-apiserver container on the control node is either not started or docker is not started. You can mimic the 6443 connection refused by doing this:

-- from the control node:

sudo systemctl stop docker

kubectl get pods

-- response:

The connection to the server 192.168.0.7:6443 was refused - did you specify the right host or port?

It can't find the API located at port 6443… Hmmm… So this demonstrates that this particular error is related to the API and the docker container running it. So if docker is down, or if anything blocking my access to docker is interrupted, it will throw this type of error.

Let's bring it back up:

sudo systemctl start docker

• give it a few minutes to start all the kubernetes system containers, remember, you don't have any application containers on the control node, usually only kube-system stuff.

• this won't affect your worker nodes or the containers on them, to them it's like a disconnect to the api and nothing more, they will remain functioning as normal unless they need something critical from the api.

kubectl get pods

-- response:

NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 42h