workload identity can work 2 different GCP project?

Question

ON GCP,I need to use 2 GCP project; One is for web-application, the other is for storing secrets for web-application ( which structure comes from google's repository

As written in README, I'll store secrets using GCP Secret Manager

This project is allocated for GCP Secret Manager for secrets shared by the organization.

procedure I'm planning

1. prj-secret : create secrets in secrets-manager
2. prj-application : read secret usingkubernetes-external-secrets

in prj-application I want to use workload identity , because I don't want to use as serviceaccountkey doc saying

What I did

1. create cluser with -workload-pool=project-id.svc.id.goog option

2. helm install kubernetes-external-secrets

3. [skip] kubectl create namespace k8s-namespace ( because I install kubernetes-external-secrets on default name space)

4. [skip] kubectl create serviceaccount --namespace k8s-namespace ksa-name ( because I use default serviceaccount with exist by default when creating GKE)

5. create google-service-account with module "workload-identity

module "workload-identity" {
   source               = "github.com/terraform-google-modules/terraform-google-kubernetes-engine//modules/workload-identity"
   use_existing_k8s_sa  = true
   cluster_name         = var.cluster_name
   location             = var.cluter_locaton
   k8s_sa_name          = "external-secrets-kubernetes-external-secrets"
   name                 = "external-secrets-kubernetes"
   roles                = ["roles/secretmanager.admin","roles/secretmanager.secretAccessor"]
   project_id           = var.project_id  #it is prj-aplication's project_id
}

• kubernetes_serviceaccount called external-secrets-kubernetes-external-secrets was already created when installing kubernetes-external-secrets with helm. and it bind k8s_sa_name &' external-secrets-kubernetes@my-project-id.iam.gserviceaccount.com, which has ["roles/secretmanager.admin","roles/secretmanager.secretAccessor"].

6. create externalsecret and apply

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: external-key-test
spec:
  backendType: gcpSecretsManager
  projectId: my-domain
  data:
    - key: key-test
      name: password

result

I got permission problem

ERROR, 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/project-id/secrets/external-key-test/versions/latest' (or it may not exist).

I already checked that, if I prj-secret and prj-application is same project, it worked.

So what I thought is, kubernetes serviceaccount (in prj-secret ) & google serviceaccount (in prj-application ) cannot bind correctly.

I wonder if someone know

• workload-identity works only in same project or not
• if it is, how can I get secret data from different project

Thank you.

Answer 1

You have an issue in your role binding I think. When you say this:

kubernetes_serviceaccount called external-secrets-kubernetes-external-secrets was already created when installing kubernetes-external-secrets with helm. and it bind k8s_sa_name &' external-secrets-kubernetes@my-project-id.iam.gserviceaccount.com, which has ["roles/secretmanager.admin","roles/secretmanager.secretAccessor"].

It's unclear.

1. external-secrets-kubernetes@my-project-id.iam.gserviceaccount.com, is created on which project? I guess in prj-application, but not clear.

---

2. I take the assumption (with the name and the link with the cluster) that the service account is created in the prj-application. you grant the role "roles/secretmanager.admin","roles/secretmanager.secretAccessor" on which resource?

• On the IAM page of the prj-application?
• On the IAM page of the prj-secret?
• On the secretId of the secret in the prj-secret?

If you did the 1st one, it's the wrong binding, the service account can only access to the secret of the prj-application, and not these of prj-secret.

Note, if you only need to access the secret, don't grand the admin role, only the accessor is required.

Answer 2

@guillaume blaquiere

Thank you so much for your kind response.

I had made serviceaccount external-secrets-kubernetes@my-project-id.iam.gserviceaccount.com which my-project-id is prj-application

Update:(v1):delete because there are not correct

So I would change it to `external-secrets-kubernetes@prj-secret.iam.gserviceaccount.com`, which has role "roles/secretmanager.admin","roles/secretmanager.secretAccessor"

About secret_id, its of prj-secret now.

---

Update:(v1) added conclusion

I understood that.

external-secrets-kubernetes@prj-application.iam.gserviceaccount.com

• must be in IAM of prj-secret PROJECT
• must have role role/secretmanager.secretAccessor

Again thank you @guillaume blaquiere

---

Update:(v1):delete because there are not correct

After changing - On the IAM page of the prj-secret, I have `external-secrets-kubernetes@prj-secret.iam.gserviceaccount.com`

But then I had problem; IAM binding looks not working.

In current structure I need to bind between

• k8s-serviceaccount (external-secrets-kubernetes-external-secrets) in prj-application
• google-serviceaccount (external-secrets-kubernetes) in prj-secret

doesn't it?

But I cannot bind it.

$ gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:prj-application.svc.id.goog[default/external-secrets-kubernetes-external-secrets]" external-secrets-kubernetes@prj-secret.iam.gserviceaccount.com

ERROR: (gcloud.iam.service-accounts.add-iam-policy-binding) NOT_FOUND: Service account projects/prj-application/serviceAccounts/external-secrets-kubernetes@prj-secret.iam.gserviceaccount.com does not exist.

Based on error, in prj-application, I don't have serviceaccount external-secrets-kubernetes@prj-secret.iam.gserviceaccount.com, (I know, because I changed as it is)

So binding k8s-serviceaccount of prj-application & google-serviceaccount of prj-secret impossible?

Answer 3

@Ryo I am still trying to understand between two project. Could you please help me with the step by step like how it really works. Could not find anything for workload identity between two projects: @guillaume blaquiere

Answer 4

Solved the workload identity between two projects: GKE Workload Identity between two projects

• prj-secret: Will have the Google Service account with all required permission.
• prj-application: Will be running the kubernetes cluster.

The link above solved the same issue between two projects where secret is in ProjectA and GKE Cluster is running in Project B