kubectl exec “错误：无法升级连接：未经授权”

Question

I was using our Kubernetes cluster, I don't think so i have changed recently after deployment but am encountering this error

Error kubectl log with verbose :

01:49:42.691510   30028 round_trippers.go:444] Response Headers:
I0514 01:49:42.691526   30028 round_trippers.go:447]     Content-Length: 12
10514 01:49:42.691537   30028 round_trippers.go:447]     Content-Type: text/plain; charset=utf-8
                                                            I0514 01:49:42.691545   30028 round_trippers.go:447]     Date: Tue, 14 May 2019 08:49:42 GMT
                                                                                                                                                        F0514 01:49:42.691976   30028 helpers.go:119] error: unable to upgrade connection: 
 Unauthorized

Kubelet running with below options :

/usr/local/bin/kubelet --logtostderr=true --v=2 --address=0.0.0.0 --node-ip=1******
--hostname-override=***** --allow-privileged=true --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --authentication-token-webhook --enforce-node-allocatable= --client-ca-file=/etc/kubernetes/ssl/ca.crt --pod-manifest-path=/etc/kubernetes/manifests --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.1 --node-status-update-frequency=10s --cgroup-driver=cgroupfs --max-pods=110 --anonymous-auth=false --read-only-port=0 --fail-swap-on=True --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf --kube-reserved cpu=200m,memory=512M --node-labels=node-role.kubernetes.io/master=,node-role.kubernetes.io/node= --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin

API running with below options :

kube-apiserver --allow-privileged=true --apiserver-count=2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --endpoint-reconciler-type=lease --insecure-port=0 --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP --runtime-config=admissionregistration.k8s.io/v1alpha1 --service-node-port-range=30000-32767 --storage-backend=etcd3 --advertise-address=******* --client-ca-file=/etc/kubernetes/ssl/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/ssl/etcd/ca.pem --etcd-certfile=/etc/kubernetes/ssl/etcd/node-bg-kub-dev-1.pem --etcd-keyfile=/etc/kubernetes/ssl/etcd/node-bg-kub-dev-1-key.pem --etcd-servers=https://*******:2379,https://********:2379,https://*****:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/ssl/sa.pub --service-cluster-ip-range=10.233.0.0/18 --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key

Answer 1

I think you messed your cert files or you played with RBAC profiles.

You can have a look at great guide by Kelsey Hightower called kubernetes-the-hard-way .

It's showing how to setup a whole cluster from beggining without any automation tools like kubeadm.

In part 04-certificate-authority - Provisioning a CA and Generating TLS Certificates . You have exampled of certs being used in Kubernetes.

The Kubelet Client Certificates

Kubernetes uses a special-purpose authorization mode called Node Authorizer, that specifically authorizes API requests made by Kubelets . In order to be authorized by the Node Authorizer, Kubelets must use a credential that identifies them as being in the system:nodes group, with a username of system:node:<nodeName> . In this section you will create a certificate for each Kubernetes worker node that meets the Node Authorizer requirements.

Once certs are generated for workers and uploaded you need to generate kubeconfig for each worker.

The kubelet Kubernetes Configuration File

When generating kubeconfig files for Kubelets the client certificate matching the Kubelet's node name must be used. This will ensure Kubelets are properly authorized by the Kubernetes Node Authorizer .

Also this case might be helpful "kubectl exec" results in "error: unable to upgrade connection: Unauthorized"

Answer 2

I got fixed this issue.

Actually "/etc/kubernetes/ssl/ca.crt" in my both masters are same but in worker nodes "/etc/kubernetes/ssl/ca.crt" is totally different. So i just copied "/etc/kubernetes/ssl/ca.crt" from master to my worker nodes and restarted kubelet in workers nodes which fixed my issue. But am not sure I did right changes for fix

I hope --client-ca-file=/etc/kubernetes/ssl/ca.crt should be same for all kubelet which is running master and workers