azure kubernetes 服务 - 私有注册表上的自签名证书

Question

I have a tunnel created between my azure subscription and my on-prem servers. ON prem we have an artifactory server that is housing all of our docker images. For all internal servers we have a company wide CA trust and all certs are generated from this.

However, when I try to deploy something to aks and reference this docker registry. I am getting a cert error because the nodes themselves do not trust the "in house" self signed cert.

Is there anyway to get the root CA chain added to the nodes? Or a way to tell the docker daemon on the aks nodes this is an insecure registry?

Answer 1

Not one hundred percent sure, but you can try to use the docker config to create the secret for image pull, the command like this:

cat ~/.docker/config.json | base64

Then create the secret like this:

apiVersion: v1
kind: Secret
metadata:
 name: registrypullsecret
data:
 .dockerconfigjson: <base-64-encoded-json-here>
type: kubernetes.io/dockerconfigjson

Use this secret in your deployment or pod as the value of imagePullSecrets . For more details, see Using a private Docker Registry with Kubernetes .

Answer 2

For the beginning I would recommend you to use curl to check connection between your azure cluster and on prem server.

Please use curl and curl -k and check if they both works(-k allow connections to SSL sites without certs, I assume it won't work, what means You don't have on prem certs on azure cluster)

If curl -k won't work then you need to copy and add certs from on prem to azure cluster.

Links which should help you do that

• https://docs.docker.com/ee/enable-client-certificate-authentication/
• https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate

And found some informations about doing that with docker daemon

• https://docs.docker.com/registry/insecure/

I hope it will help you. Let me know if you have any more questions.

Answer 3

It looks like you are having the same problem described here: https://github.com/kubernetes/kubernetes/issues/43924 .

This solution should probably work for you:

As far as I remember this was a docker issue, not a kubernetes one. Docker does not use linux's ca certs. Nobody knows why.

You have to install those certs manually (on every node that could spawn those pods) so that docker can use them:

/etc/docker/certs.d/mydomain.com:1234/ca.crt

This is a highly annoying issue as you have to butcher your nodes after bootstrapping to get those certs in there. And kubernetes spawns nodes all the time. How this issue has not been solved yet is a mystery to me. It's a complete showstopper IMO.

Then it's just a question of how to run this for every node. You could do that with a DaemonSet which runs a script from a ConfigMap, as described here: https://cloud.google.com/solutions/automatically-bootstrapping-gke-nodes-with-daemonsets . That article refers to a GitHub project https://github.com/GoogleCloudPlatform/solutions-gke-init-daemonsets-tutorial . The magic is in the DaemonSet.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-initializer
  labels:
    app: default-init
spec:
  selector:
    matchLabels:
      app: default-init
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: node-initializer
        app: default-init
    spec:
      volumes:
      - name: root-mount
        hostPath:
          path: /
      - name: entrypoint
        configMap:
          name: entrypoint
          defaultMode: 0744
      initContainers:
      - image: ubuntu:18.04
        name: node-initializer
        command: ["/scripts/entrypoint.sh"]
        env:
        - name: ROOT_MOUNT_DIR
          value: /root
        securityContext:
          privileged: true
        volumeMounts:
        - name: root-mount
          mountPath: /root
        - name: entrypoint
          mountPath: /scripts
      containers:
      - image: "gcr.io/google-containers/pause:2.0"
        name: pause

You could modify the script that is in the ConfigMap to pull your cert and put it in the correct directory.