kubectl代理无法与API服务器通信

Question

I'm having a problem with kubectl proxy on a fresh install.

When I browse to http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ I get a 503 response. It seems that the proxy isn't able to reach the kubernetes API even though other commands can.

Kubernetes runs in DC/OS with package 1.3.1-1.10.8. Both kubectl and Kubernetes are version 1.10.8. There's a load balancer configured in dc/os to expose the API.

The LB definition is from the kubernetes on dcos help page . I added "HAPROXY_0_VHOST": "k8s-proxy.dcos.<domain>.com" to the labels.

$ kubectl cluster-info
Kubernetes master is running at https://k8s-proxy.dcos.<domain>.com
KubeDNS is running at https://k8s-proxy.dcos.<domain>.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

I ran kubectl proxy in verbose output mode to see what call it was trying to make. It got a 503 response.

$ kubectl proxy --insecure-skip-tls-verify=true --alsologtostderr=true -v=99
I0327 12:26:45.461259   19980 loader.go:357] Config loaded from file U:\/.kube/config
Starting to serve on 127.0.0.1:8001
I0327 12:26:56.200819   19980 proxy_server.go:98] /api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ matched ^.*
I0327 12:26:56.200819   19980 proxy_server.go:98] localhost matched ^localhost$
I0327 12:26:56.200819   19980 proxy_server.go:138] Filter accepting GET /api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ localhost
I0327 12:26:56.200819   19980 upgradeaware.go:237] Request was not an upgrade
I0327 12:26:56.200819   19980 round_trippers.go:387] curl -k -v -XGET  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Cache-Control: max-age=0" -H "User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" -H "Authorization: Bearer <my_token>" -H "X-Forwarded-For: 127.0.0.1" -H "Accept-Language: en-US,en;q=0.9" -H "Dnt: 1" -H "Accept-Encoding: gzip, deflate, br" -H "Upgrade-Insecure-Requests: 1" https://k8s-proxy.dcos.<domain>.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
I0327 12:26:56.313141   19980 round_trippers.go:406] GET https://k8s-proxy.dcos.<domain>.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ 503 Service Unavailable in 112 milliseconds
I0327 12:26:56.313141   19980 round_trippers.go:412] Response Headers:
I0327 12:26:56.313141   19980 round_trippers.go:415]     Cache-Control: no-cache
I0327 12:26:56.313141   19980 round_trippers.go:415]     Content-Type: text/html

In the same shell I tried running the curl that the proxy runs. It got a 200 instead of a 503.

$ curl -k -v -XGET  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Cache-Control: max-age=0" -H "User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" -H "Authorization: Bearer <my_token>" -H "X-Forwarded-For: 127.0.0.1" -H "Accept-Language: en-US,en;q=0.9" -H "Dnt: 1" -H "Accept-Encoding: gzip, deflate, br" -H "Upgrade-Insecure-Requests: 1" https://k8s-proxy.dcos.<domain>.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

[...]
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: no-store
< Content-Encoding: gzip
< Content-Type: text/html; charset=utf-8
< Date: Wed, 27 Mar 2019 19:30:24 GMT
< Last-Modified: Fri, 24 Aug 2018 05:39:29 GMT
< Content-Length: 529
[...]

I expected to be able to access my cluster but valid requests are returning a 503. Other kubectl commands work fine. This is not a problem that's specific to the dashboard.

Answer 1

The most common issue, while deploying dashboard is missing to grant privileges to the service account to manage secretes in kube-system namespace. More details here

This is causing dashboard pod to fall into crashloopbackoff and consequently, lack of endpoint for your dashboard service.

So the first step to troubleshoot it is to check endpoints

kubectl get ep -n kube-system kubernetes-dashboard