Helm upgrade is making deployment failure

Question

We configured csi driver in our cluster for secret management and used below secret providerclass template to automatically assign secrets to the deployments env variable. The above setup is working fine.

But 2 things where i have issue. Whenever new changes done to the secret, say if added a new secret to the yaml and keyvault, the next release will fail with helm upgrade command, stating speified secret is not find.

So inorder to solve this, I have to uninstall all helm release and need to install the helm release again, which meant down time. How I can I achieve this scenario without any down time.

Secondly, is there any recommended way to restart the Pods when the secret template changes:

values.yaml for MyAppA

keyvault:
  name: mykv
  tenantId: ${tenantId}$
  clientid: "#{spid}#"
  clientsecret: "#{spsecret}#"
  secrets:
    - MyAPPA_SECRET1_NAME1
    - MyAPPA_SECRET2_NAME2
    - MyAPPA_SECRET3_NAME3

deployment.yaml, ENV part is as below

    {{- if eq .Values.keyvault.enabled true }}
    {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}
      - name: {{ . }}
        valueFrom:
          secretKeyRef:
            name: {{ $.Release.Name }}-kvsecret
            key: {{ . }}
      {{- end }}
      {{- end }}
      volumeMounts: 
      - name:  {{ $.Release.Name }}-volume
        mountPath: '/mnt/secrets-store'
        readOnly: true
  volumes:
    - name: {{ $.Release.Name }}-volume
      csi:
        driver: 'secrets-store.csi.k8s.io'
        readOnly: true
        volumeAttributes:
          secretProviderClass: {{ $.Release.Name }}-secretproviderclass
        nodePublishSecretRef:
          name: {{ $.Release.Name }}-secrets-store-creds
          
          

secretProviderClass yaml file is as below.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: {{ $.Release.Name }}-secretproviderclass
  labels:
    app: {{ $.Release.Name }}
    chart: "{{ $.Release.Name }}-{{ .Chart.Version }}"
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  provider: azure
  secretObjects:
  - data:
      {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}
    - key: {{ . }}
      objectName: {{ $.Release.Name | upper }}-{{ . }} 
      {{- end }}      
    secretName: {{ $.Release.Name }}-kvsecret
    type: opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: ""
    keyvaultName: {{ .Values.keyvault.name | default "mydev-kv" }}   
    objects: |
      array:
        {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}        
        - |
          objectName: {{ $.Release.Name | upper }}-{{ . }}
          objectType: secret
        {{- end }}
    tenantId: {{ .Values.keyvault.tenantid }}
{{- end }}
{{- end -}}
{{- define "commonobject.secretproviderclass" -}}
{{- template "commonobject.util.merge" (append . "commonobject.secretproviderclass.tpl") -}}
{{- end -}}

Answer 1

Here is mentioned a similar issue: https://github.com/helm/helm/issues/11104

Unfortunately no useful answer there yet. I have the same problem. After updating SecretClassProvider (adding or removing secret data) "helm upgrade" doesn't update k8s secret resources.

Answer 2

The problem is not in the "helm upgrade" command. I discovered this is a limitation of a CSI driver or SecretProviderClass. When the deployment is already created, the ServiceProviderClass resource is updated but the "SecretProviderClassPodStatuses" is not, so secrets are not updated.

Two potential solutions to update secrets:

• delete secret and restart/recreate pod => this works but it sounds more like a workaround than an actual solution
• set enableSecretRotation to true => it has been implemented in a CSI driver recently and it's in an 'alpha' version, I couldn't make it work yet.

https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html