如何为 Azure Keyvault + SecretProviderClass + imagePullSecrets+ Private docker 存储库配置部署文件

Question

How to configure deployment file for the combination of Azure Keyvault + SecretProviderClass + imagePullSecrets+ Private docker repository.

We have private docker repository to maintain images, now we have a requirement maintaining the credentials of that Docker repository in Azure key vault, import it into AKS using SecretProviderClass, use that secret under 'imagePullSecrets'

# This is a SecretProviderClass example using system-assigned identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-system-harbor
spec:
  provider: azure
  secretObjects:
  - secretName: harborcredentialvault
    data:
    - key: harborcredentialvaultkey
      objectName: harborcredentialvault
    type: kubernetes.io/dockerconfigjson
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"    # Set to true for using managed identity
    userAssignedIdentityID: ""      # If empty, then defaults to use the system assigned identity on the VM
    keyvaultName: "<Keyvault name>"
    cloudName: ""                   # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: harborcredentialvault
          objectType: secret        # object types: secret, key, or cert
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
    tenantId: "<tenant ID>"           # The tenant ID of the key vault

        - name: harborcredentialvault
          valueFrom: 
            secretKeyRef:
              name: keyvault-secret
              key: harborcredentialvaultkey
      imagePullSecrets:
       - name: ${harborcredentialvault}
        volumeMounts:
         - mountPath: "/mnt/secrets-store"
           name: secrets-store01-inline
           readOnly: true
       - name: secrets-store01-inline
         csi:
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes:
             secretProviderClass: "azure-kvname-system-harbor"

Answer 1

As you do not provided a real question or an error im will be a bit general:

For the AKS/KeyVault integration it is important to understand that you are accessing the Key Vault with the Kubelet Identity of the Nodepool and not with the Managed Identity of the AKS as described here . So if you are using Managed Identity userAssignedIdentityID should not be empty.

So we need to give the Kubelet Identity access to the Key Vault, for example like this:

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

The result of $KUBE_ID needs to be also added the the SecretProviderClass :

userAssignedIdentityID: "RESULT"

From this official example here your SecretProviderClass looks good for this use case.

This would be the pod config:

spec:
  containers:
  - name: demo
    image: demo
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  imagePullSecrets:
    - name: harborcredentialvault
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "azure-kvname-system-harbor"

This should sync the Key Vault secret to a Kube.netes secret. Here is also the documentation .

One thing you should consider is = The secrets will only sync once you start a pod mounting the secrets. Solely relying on the syncing with Kube.netes secrets feature thus does not work. The secrets will only sync once you start a pod mounting the secrets. Solely relying on the syncing with Kube.netes secrets feature thus does not work.

That being said you maybe would need another pod with a public image to sync your private pull secrets for your cluster bcs your pod would not start as it can not pull the image from your private registry.

Answer 2

@Philip Welz answer helped me to find the below solution

SecretProviderClass sample yaml

# This is a SecretProviderClass example using system-assigned identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-system-harbor
spec:
  provider: azure
  secretObjects:
    - secretName: dockerconfig
      type: kubernetes.io/dockerconfigjson
      data:
        - objectName: harborcredentialvault
          key: .dockerconfigjson
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"    # Set to true for using managed identity
    userAssignedIdentityID: ""      # If empty, then defaults to use the system assigned identity on the VM
    keyvaultName: "<Keyvault name>"
    cloudName: ""                   # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: harborcredentialvault
          objectType: secret        # object types: secret, key, or cert
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
    tenantId: "<tenant ID>"           # The tenant ID of the key vault

Deployment sample yaml file

spec:
  containers:
  - name: demo
    image: demo
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  imagePullSecrets:
    - name: dockerconfig
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "azure-kvname-system-harbor"

Create Secret in Keyvault, make sure value should be in below JSON format

Key: harborcredentialvault
Value: {
"auths": {
"dockerwebsite.com": {
"username": "username",
"password": "password"
}
}

}