Kubernetes：kubectl 顶级节点/Pod 不工作

Question

When I do kubectl top pods/nodes I am getting the following error:

metrics not available yet

When I check the logs I am getting

$ kubectl logs metrics-server-7df4c4484d-h75wr -n kube-system -c metrics-server

I1008 12:02:55.159069       1 serving.go:273] Generated self-signed cert(apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/10/08 12:02:56 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/10/08 12:02:56 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1008 12:02:56.358063       1 serve.go:96] Serving securely on [::]:443
E1008 12:03:04.225519       1 reststorage.go:101] unable to fetch node metrics for node "hpa-test": no metrics known for node "hpa-test"
E1008 12:03:07.619489       1 reststorage.go:101] unable to fetch node metrics for node "hpa-test": no metrics known for node "hpa-test"

Also, I am able to ping to hpa-test node from:

$ kubectl exec -it  metrics-server-7df4c4484d-h75wr -n kube-system sh

Also, I have tried looking for solution everywhere but nothing fixed the issue

Answer 1

There are two ways to fix this problem:

1) using heapster : installing heapster will allow 'kubectl top nodes' to work out of the box. However heapster has been deprecated, so you probably should switch to metrics-server.

2) using metrics-server : unfortunately it may not work out of the box when installing metrics-server... in case it doesn't, you need to update the end of the metrics-server-deployment.yaml (1.8+) file you used for installation and add a command section with the right parameters as follow:

containers:
- name: metrics-server
  image: k8s.gcr.io/metrics-server-amd64:v0.3.1
  imagePullPolicy: Always
  volumeMounts:
  - name: tmp-dir
    mountPath: /tmp

  command:
      - /metrics-server
      - --kubelet-insecure-tls
      - --kubelet-preferred-address-types=InternalIP

then simply apply the changes:

kubectl apply -f metrics-server-deployment.yaml

you should then be able to get results with

kubectl top nodes

and

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

Answer 2

Need add flags for metrics-sever:

--kubelet-insecure-tls=true
--kubelet-port={YOU_KUBELET_PORT}
--kubelet-preferred-address-types=InternalIP
--v=5
--logtostderr

Answer 3

These issue arises because you don't have metric server enabled in your cluster

The top command will only be running if the metric server installed on your cluser

Download metric server in Kubernetes API from the git

Step:1 git clone https://github.com/kubernetes-incubator/metrics-server.git

Step:2 cd metrics-server/

Step:3 Deploy

Kubernetes 1.7 kubectl create -f deploy/1.7/

Kubernetes 1.8

kubectl create -f deploy/1.8+/

Setp :4 Check status of metric server pod

 kubectl get po -n kube-system |grep metrics

Step5: Now your top command will be running

kubectl top pod --all-namespaces

Answer 4

Edit the deployment of metrics-server and add the following to arguments to its container.

--kubelet-insecure-tls
--kubelet-preferred-address-types=InternalIP

For example,

kind: Deployment
metadata:
  name: metrics-server
  ...
spec:
  template:
    spec:
      ...
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        args:
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP