Kubernetes Dashboard an error on the server ("unknown") has prevented the request from succeeding

Question

After getting my k8s cluster up and going I faithfully deployed the following WebUI dashboard using the command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml

When I try to access it I get the following error:

Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper)

If I get all the services I get:

k get services --all-namespaces
NAMESPACE              NAME                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
default                kubernetes                  ClusterIP   10.96.0.1     <none>        443/TCP         8d
kube-system            kube-dns                    ClusterIP   10.96.0.10    <none>        53/UDP,53/TCP   8d
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.96.0.65    <none>        8000/TCP        6m10s
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.96.0.173   <none>        443/TCP         6m10s

Can someone shed some light? what am I missing?

More Info: In the dashboard yaml I found these roles:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
   
 map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

    ---
    
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
    rules:
      
      - apiGroups: ["metrics.k8s.io"]
        resources: ["pods", "nodes"]
        verbs: ["get", "list", "watch"]
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kubernetes-dashboard
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: kubernetes-dashboard
    subjects:
      - kind: ServiceAccount
        name: kubernetes-dashboard
        namespace: kubernetes-dashboard
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: kubernetes-dashboard
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: kubernetes-dashboard
    subjects:
      - kind: ServiceAccount
        name: kubernetes-dashboard
        namespace: kubernetes-dashboard

Looks like the kube.netes-dashboard user has access to the metrics service I might be wrong

Answer 1

It looks like the kube.netes-dashboard's serviceaccount doesn't have access to all kube.netes resources (in particular, it can't access the metric server service).

To fix this you should create a new ServiceAccount for the dashboard and give it more permissions.

Here's one that I found on another similar post ( be careful since it will give admin privileges to the dashboard, and whoever uses it will be able to destroy/create new or existing resources on your kube.netes cluster ):

   apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
   name: kubernetes-dashboard
   labels:
       k8s-app: kubernetes-dashboard
roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

If you don't have a cluster-admin ServiceAccount, create one following this template:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

Admin ClusterRole:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: admin
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
    nonResourceURLs: ["*"]