是否可以在Kubernetes中启动自签名Docker注册表并将其他服务用作该注册表来获取其映像？

Question

Problem Statement

• I want to deliver a private registry which all the images I need for my product bundled in them ( Yes, It will be fat image, but I am fine with that)
• I would manually upload this image in some way
• I would run the docker private registry as a service in Kubernetes (probably in some namespace)
• When other services/deployments (in the same namespace as registry) happen in Kubernetes, they should refer to this registry using a consistent name

Constraints

• We want registry to be exposed only to the cluster and not outside
• We want to use self signed certificate and not signed by CA

I followed some instructions from these links (do not know whether it was a right thing to do)

• https://kubernetes.io/docs/concepts/cluster-administration/certificates/
• https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificate-signing-request-object-to-send-to-the-kubernetes-api

Create a certificate signed through Kubernetes

1. Create a server.key

2. Create a csr.info

[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C = US
ST = oh
L = cincinnati
O = engg
OU = prod
CN = prateek.svc.cluster.local

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = registry.prateek.svc.cluster.local

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names

3. Created the server.csr (openssl req -new -key server.key -out server.csr -config csr.conf)

4. Create the CertificateSigningRequest in K8s

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: registry.prateek
spec:
groups:
- system:authenticated
request: $(cat server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF

5. Checked if the CSR exists

kubectl describe csr registry.prateek
Name: registry.prateek
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certificates.k8s.io/v1beta1","kind":"CertificateSigningRequest","metadata":{"annotations":{},"name":"registry.prateek","namespace":""},"spec":{"groups":["system:authenticated"],"request":"LS0sdfsfsdsfd=","usages":["digital signature","key encipherment","server auth"]}}

CreationTimestamp: Thu, 11 Apr 2019 11:15:42 -0400
Requesting User: docker-for-desktop
Status: Pending
Subject:
Common Name: prateek.svc.cluster.local
Serial Number:
Organization: engg
Organizational Unit: prod
Country: US
Locality: cincinnati
Province: oh
Subject Alternative Names:
DNS Names: registry.prateek.svc.cluster.local
Events: <none>

6. Approved the CSR : kubectl certificate approve registry.prateek

Start the registry internal service

7. Added cert and key to the kind: Secret

registry-secret.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: prateek
  labels:
      app: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      containers:
        - name: registry
          image: prateek/registry
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 443
          env:
            - name: REGISTRY_HTTP_ADDR
              value: "0.0.0.0:443"
            - name: REGISTRY_HTTP_TLS_CERTIFICATE
              value: "/certs/certificate"
            - name: REGISTRY_HTTP_TLS_KEY
              value: "/certs/key"
          volumeMounts:
            - name: cert-files
              mountPath: /certs
      volumes:
        - name: cert-files
          secret:
            secretName: registry-credentials

8. Create registry deployment and service (using those secrets) registry-deployment.yml

apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: prateek
spec:
  selector:
    app: registry
  ports:
  - protocol: TCP
    port: 443
    targetPort: 443
  type: LoadBalancer

registry-service.yml

 apiVersion: v1 kind: Service metadata: name: registry namespace: prateek spec: selector: app: registry ports: - protocol: TCP port: 443 targetPort: 443 type: LoadBalancer 

Test regsitry service is up

9. Tried to this the registry endpoint thru a test pod. I had image of this test pod loaded in docker already.

 curl https://registry.prateek.svc.cluster.local/v2/_catalog -k {"repositories":["prateek/echo"]} 

Deployment using image from the registry service

10. Tried deployment with image: registry.prateek/prateek/echo:latest

Normal Pulling 10s (x2 over 25s) kubelet, docker-for-desktop pulling image "registry.prateek/prateek/echo:latest"
Warning Failed 10s (x2 over 25s) kubelet, docker-for-desktop Failed to pull image "registry.prateek/prateek/echo:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.prateek/v2/: Service Unavailable 

11. the deployment gives the error

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: cequence
  labels:
      app: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: registry.prateek.svc.cluster.local/prateek/echo:latest
        imagePullPolicy: IfNotPresent
        ports:
         - containerPort: 5678
        args: ["-text=hello"]

12. Changed deployment to have image: registry.prateek.svc.cluster.local/prateek/echo:latest

Warning Failed 1s kubelet, docker-for-desktop Failed to pull image "registry.prateek.svc.cluster.local/prateek/echo:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.prateek.svc.cluster.local/v2/: Service Unavailable

13. get the similar error

 Warning Failed 1s kubelet, docker-for-desktop Failed to pull image "registry.prateek.svc.cluster.local/prateek/echo:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.prateek.svc.cluster.local/v2/: Service Unavailable 

I do not that this is even possible. Run a docker registry as a service and point other service in the namespace to use that registry deployment in the cluster. Any suggestion is welcome

Answer 1

The container daemon is running outside of kubernetes.

Therefore, if you want to pull the image, you need to make sure that the registry is reachable from the node directly, without using kubernetes mechanisms like a service. (Not like you tested it in step 9 through a pod, you must be able to work directly on the node!)

The usual options are to create a DNS entry or hosts.txt entry to point to a node where either through a hostPort (container) or nodePort (service) the registry is accessible or you use an appropriate ingress.