AKS 中的 Cert-manager ClusterIssuer 問題

Question

我在 clusterissuer（證書管理器版本 1.7.1）中收到此錯誤：

“獲取 CA 頒發者的密鑰對時出錯：解碼證書 PEM 塊時出錯”

我將 ca.crt、tls.crt 和 tls.key 存儲在 Azure 的 Key Vault 中。

kubectl 描述 clusterissuer ca-issuer

  Ca:
    Secret Name:  cert-manager-secret
Status:
  Conditions:
    Last Transition Time:  2022-02-25T11:40:49Z
    Message:               Error getting keypair for CA issuer: error decoding certificate PEM block
    Observed Generation:   1
    Reason:                ErrGetKeyPair
    Status:                False
    Type:                  Ready
Events:
  Type     Reason         Age                  From          Message
  ----     ------         ----                 ----          -------
  Warning  ErrGetKeyPair  3m1s (x17 over 58m)  cert-manager  Error getting keypair for CA issuer: error decoding certificate PEM block
  Warning  ErrInitIssuer  3m1s (x17 over 58m)  cert-manager  Error initializing issuer: error decoding certificate PEM block

kubectl 獲取集群發行者

NAME        READY   AGE
ca-issuer   False   69m 

• 這是 clusterissuer yaml 文件：

ca-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: cert-manager-secret

這是用於檢索 ca.crt、tls.crt 和 tls.key 的 KeyVault yaml 文件

keyvauls.yaml

apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akscacrt
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akscacrt # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: ca.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akstlscrt
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akstlscrt # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: tls.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akstlskey
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akstlskey # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: tls.key # key to store object value in kubernetes secret
---

這些是使用的證書：

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: argocd-xx
  namespace: argocd
spec:
  secretName: argocd-xx
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  commonName: "argocd.xx"
  dnsNames:
    - "argocd.xx"
  privateKey:
    size: 4096
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sonarqube-xx
  namespace: sonarqube
spec:
  secretName: "sonarqube-xx"
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  commonName: "sonarqube.xx"
  dnsNames:
    - "sonarqube.xx"
  privateKey:
    size: 4096

我可以看到我可以從密鑰庫中檢索證書的秘密：

kubectl get secret -n cert-manager cert-manager-secret -o yaml

apiVersion: v1
data:
  ca.crt: XXX
  tls.crt: XXX
  tls.key: XXX

另外，另一件奇怪的事情是我在之前部署的 sonarqube/argocd 命名空間中獲得了其他秘密，但我的部署文件中不再有這些秘密。 我無法刪除它們，當我嘗試刪除它們時，它們會自動重新創建。 看起來它們存儲在某種緩存中。 我還嘗試刪除命名空間 akv2k8s/cert-manager 並刪除 cert-manager/akv2k8s 控制器並重新安裝它們，但在重新安裝和應用部署后出現同樣的問題......

kubectl get secret -n sonarqube

NAME                                      TYPE                                  DATA   AGE
cert-manager-secret                       Opaque                                3      155m
default-token-c8b86                       kubernetes.io/service-account-token   3      2d1h
sonarqube-xx-7v7dh   Opaque                                1      107m
sql-db-secret                             Opaque                                2      170m

kubectl get secret -n argocd   
NAME                                   TYPE                                  DATA   AGE
argocd-xx-7b5kb   Opaque                                1      107m
cert-manager-secret-argo               Opaque                                3      157m
default-token-pjb4z                    kubernetes.io/service-account-token   3      3d15h

kubectl 描述證書 sonarqube-xxx -n sonarqube

Status:
  Conditions:
    Last Transition Time:        2022-02-25T11:04:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2022-02-25T11:04:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  sonarqube-xxx-7v7dh
Events:                          <none>

任何的想法？

謝謝。

Answer 1

我想通了只是上傳證書信息ca.crt 。 純文本的tls.crt和tls.key ，在 Azure 的 Key Vault 機密中沒有 BASE64 編碼。

當 AKV2K8S 從 Key Vault 中檢索機密並存儲在 Kube.netes 中時，它會自動編碼為 BASE64。

問候，