use AWS Secrets & Configuration Provider for EKS: Error from server (BadRequest)

Question

I'm following this AWS documentation which explains how to properly configure AWS Secrets Manager to let it works with EKS through Kube.netes Secrets.

I successfully followed step by step all the different commands as explained in the documentation.

The only difference I get is related to this step where I have to run:

kubectl get po --namespace=kube-system 

The expected output should be:

csi-secrets-store-qp9r8         3/3     Running   0          4m
csi-secrets-store-zrjt2         3/3     Running   0          4m

but instead I get:

csi-secrets-store-provider-aws-lxxcz               1/1     Running   0          5d17h
csi-secrets-store-provider-aws-rhnc6               1/1     Running   0          5d17h
csi-secrets-store-secrets-store-csi-driver-ml6jf   3/3     Running   0          5d18h
csi-secrets-store-secrets-store-csi-driver-r5cbk   3/3     Running   0          5d18h

As you can see the names are different, but I'm quite sure it's ok:-)

The real problem starts here in step 4 : I created the following YAML file (as you ca see I added some parameters):

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
 name: aws-secrets
spec:
 provider: aws
 parameters:
   objects:  |
     - objectName: "mysecret"
       objectType: "secretsmanager"

And finally I created a deploy (as explain here in step 5 ) using the following yaml file:

# test-deployment.yaml
kind: Pod
apiVersion: v1
metadata:
  name: nginx-secrets-store-inline
spec:
  serviceAccountName: iamserviceaccountforkeyvaultsecretmanagerresearch
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - name: mysecret-volume
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: mysecret-volume
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"

After the deployment through the command:

kubectl apply -f test-deployment.yaml -n mynamespace

The pod is not able to start properly because the following error is generated:

Error from server (BadRequest): container "nginx" in pod "nginx-secrets-store-inline" is waiting to start: ContainerCreating

But, for example, if I run the deployment with the following yaml the POD will be successfully created

# test-deployment.yaml

kind: Pod
apiVersion: v1
metadata:
  name: nginx-secrets-store-inline
spec:
  serviceAccountName: iamserviceaccountforkeyvaultsecretmanagerresearch
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - name: keyvault-credential-volume
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: keyvault-credential-volume
      emptyDir: {} # <<== !! LOOK HERE !!

as you can see I used

emptyDir: {}

So as far I can see the problem here is related to the following YAML lines:

      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"

To be honest it's even not clear in my mind what's happing here. Probably I didn't properly enabled the volume permission in EKS?

Sorry but I'm a newbie in both AWS and Kube.netes configurations. Thanks for you time

--- NEW INFO ---

If I run

kubectl describe pod nginx-secrets-store-inline -n mynamespace

where nginx-secrets-store-inline is the name of the pod, I get the following output:

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    30s                default-scheduler  Successfully assigned mynamespace/nginx-secrets-store-inline to ip-10-0-24-252.eu-central-1.compute.internal
  Warning  FailedMount  14s (x6 over 29s)  kubelet            MountVolume.SetUp failed for volume "keyvault-credential-volume" : rpc error: code = Unknown desc = failed to get secretproviderclass mynamespace/aws-secrets, error: SecretProviderClass.secrets-store.csi.x-k8s.io "aws-secrets" not found

Any hints?

Answer 1

Finally I realized why it wasn't working. As explained here , the error:

  Warning  FailedMount  3s (x4 over 6s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to get secretproviderclass default/azure, error: secretproviderclasses.secrets-store.csi.x-k8s.io "azure" not found

is related to namespace:

The SecretProviderClass being referenced in the volumeMount needs to exist in the same namespace as the application pod.

So both the yaml file should be deployed in the same namespace (adding, for example, the -n mynamespace argument). Finally I got it working!