x509：由未知权威度量服务器签署的证书

Question

I am new to kubernetes and I finally realized how to launch the metrics-server as documented kubernetes-sigs/metrics-server . In case that someone else wonders you need to deploy on Master node and also have minimum one worker in the cluster.

So I get this error:

E0818 15:25:22.835094       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:<hostname-master>: unable to fetch metrics from Kubelet <hostname-master> (<hostname-master>): Get https://<hostname-master>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:<hostname-worker>: unable to fetch metrics from Kubelet <hostname-worker> (<hostname-worker>): Get https://<hostname-worker>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]

I am using my own CAs (not self signed) and I have modified the components.yml file (sample):

args:
  - --cert-dir=/tmp/metricsServerCas
  - --secure-port=4443
  - --kubelet-preferred-address-types=Hostname

I know that I can disable the tls by using this flag --kubelet-insecure-tls I have already tried it. I want to use my own CAs for extra security.

I have see other many relevant questions (few samples) eg:

x509 certificate signed by unknown authority- Kubernetes and kubectl unable to connect to server: x509: certificate signed by unknown authority

Although that I have applied chown already my $HOME/.kube/config still I see this error.

Where am I going wrong?

Update: On the worker I am creating a directory eg /tmp/ca and I add the ca file(s) in the directory.

I am not really good yet with the mounting points and I assume that I am doing something wrong. The default syntax of the images can be found here kubernetes-sigs/metrics-server/v0.3.7 (see components.yml file).

I tried to create a directory on my worker eg /tmp/ca and I modified the flag --cert-dir=/tmp/ca and mountPath: /tmp/ca

When I am deploying the file eg:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

I keep getting the error from the metrics-server-xxxx:

panic: open /tmp/client-ca-file805316981: read-only file system

Although I have given full access to the directory eg:

$ ls -la /tmp/ca
total 8
drwxr-xr-x.  2 user user   20 Aug 19 16:59 .
drwxrwxrwt. 18 root        root        4096 Aug 19 17:34 ..
-rwxr-xr-x.  1 user user 1025 Aug 19 16:59 ca.crt

I am not sure where I am going wrong.

How is meant to be configured so someone can use non self signed certificates? I can see that most people are using non SSL which I would like to avoid.

Sample of my args in the image:

spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp/ca
          - --secure-port=4443
          - --kubelet-preferred-address-types=Hostname
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp/ca
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"

Update 2: Adding curl command from Master to Worker including error output:

$ curl --cacert /etc/kubernetes/pki/ca.crt https://node_hostname:10250/stats/summary?only_cpu_and_memory=true
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Answer 1

Posting this answer as a community wiki to give better visibility as the solution was posted in the comments.

The version that I used before was 1.18.2 and metrics server v0.3.6. Deployment was through kubeadm. Yes all requirements was exactly as the metrics-server/requirements. The good news is that I got it running by upgrading my k8s version on 1.19.0 and using the latest version v0.3.7. It works with self signed certificates.

The issue was resolved by upgrading:

• Kubernetes : 1.18.2 -> 1.19.0
• Metrics-server : 0.3.6 -> 0.3.7

This upgrade allowed to run metrics-server with tls enabled (self-signed certificates).

---

Additional resources that could help when deploying metrics-server with tls :

• Github.com: Kubernetes-sigs: Metrics-server: FAQ: How to run metrics-server-securely

How to run metrics-server securely? Suggested configuration:

• Cluster with RBAC enabled
• Kubelet read-only port port disabled
• Validate kubelet certificate by mounting CA file and providing --kubelet-certificate-authority flag to metrics server
• Avoid passing insecure flags to metrics server (--deprecated-kubelet-completely-insecure, --kubelet-insecure-tls)
• Consider using your own certificates (--tls-cert-file, --tls-private-key-file)

• Github.com: Metrics-server: x509: certificate signed by unknown authority
• Ftclausen.github.io: Setting up K8S with metrics-server

Answer 2

Create a configmap to store the ca certificate which was used to generate kubelet serving certificate.

kubectl -n kube-system create configmap ca --from-file=ca.crt=/etc/kubernetes/pki/ca.crt -o yaml

Then use volumeMounts to use it in metrics server pod

spec:
  volumes:
  - emptyDir: {}
    name: tmp-dir
  - configMap:
      defaultMode: 420
      name: ca
    name: ca-dir
  containers:
    args:
      - --cert-dir=/tmp
      - --secure-port=4443
      - --kubelet-certificate-authority=/ca/ca.crt
      - --kubelet-preferred-address-types=Hostname
    volumeMounts:
    - mountPath: /tmp
      name: tmp-dir
    - mountPath: /ca
      name: ca-dir

You can follow the same approach and use --tls-cert-file and --tls-private-key-file for using your own certificate instead of self signed certificate.