My kubernetes cluster IP address changed and now kubectl will no longer connect

Question

• Running under Ubuntu I used kubeadm init to setup my cluster (master node) and copied over the /etc/kubernetes/admin.conf $HOME/.kube/config and all was well when using kubectl .
• However after a reboot my master node has had an IP address change which is not the same as what is in $HOME/.kube/config so now I can no longer connect kubectl

So how do I regenerate the admin.conf now that I have a new IP address? Running kubeadm init will just kill everything which is not what I want.

Answer 1

The following command can be used to regenerate admin.conf

kubeadm alpha phase kubeconfig admin --apiserver-advertise-address <new_ip>

However, if you use an IP instead of a hostname, your API-server certificate will be invalid. So, either regenerate your certs ( kubeadm alpha phase certs renew apiserver ), use hostnames instead of IPs or add the insecure --insecure-skip-tls-verify flag when using kubectl

Answer 2

I found this solution on the internet and it works for me:

    systemctl stop kubelet docker
    cd /etc/
    mv kubernetes kubernetes-backup
    mv /var/lib/kubelet /var/lib/kubelet-backup
    mkdir -p kubernetes
    cp -r kubernetes-backup/pki kubernetes
    rm kubernetes/pki/{apiserver.*,etcd/peer.*}
    systemctl start docker
    kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcd
    #Run "kubeadm reset" on all nodes if was this error "error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists"
    cp kubernetes/admin.conf ~/.kube/config
    kubectl get nodes --sort-by=.metadata.creationTimestamp
    kubectl delete node $(kubectl get nodes -o jsonpath='{.items[(@.status.conditions[0].status=="Unknown")].metadata.name}')
    kubectl get pods --all-namespaces

After These, Join your Slaves to Master. Reference: https://medium.com/@juniarto.samsudin/ip-address-changes-in-kubernetes-master-node-11527b867e88

Answer 3

You do not want to use kubeadm reset . That will reset everything and you would have to start configuring your cluster again.

Well, in your scenario, please have a look on the steps below:

1. nano /etc/hosts (update your new IP against YOUR_HOSTNAME )

2. nano /etc/kubernetes/config (configuration settings related to your cluster) here in this file look for the following params and update accordingly

   KUBE_MASTER="--master=http://YOUR_HOSTNAME:8080"

   KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME:2379" #2379 is default port

3. nano /etc/etcd/etcd.conf ( conf related to etcd )

   KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME/WHERE_EVER_ETCD_HOSTED:2379"

   2379 is default port for etcd . and you can have multiple etcd servers defined here comma separated

4. Restart kubelet , apiserver , etcd services.

It is good to use hostname instead of IP to avoid such scenarios.

Hope it helps!