无需凭据/身份验证即可从 GKE Pod 进行 GCS 读写访问

Question

We created the GKE cluster with the public endpoint. The service account of the GKE cluster and Node pool has the following roles.

"roles/compute.admin",
"roles/compute.viewer",
"roles/compute.securityAdmin",
"roles/iam.serviceAccountUser",
"roles/iam.serviceAccountAdmin",
"roles/resourcemanager.projectIamAdmin",
"roles/container.admin",
"roles/artifactregistry.admin",
"roles/storage.admin"

The node pool of the GKE cluster has the following OAuth scopes

"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/devstorage.read_write",

The private GCS bucket has the same service account as the principal, with the storage admin role.

When we try to read/write in this bucket from a GKE POD, we get the below error.

# Read
AccessDeniedException: 403 Caller does not have storage.objects.list access to the Google Cloud Storage bucket

# Write
AccessDeniedException: 403 Caller does not have storage.objects.create access to the Google Cloud Storage object

We also checked this thread but the solution was credential oriented and couldn't help us. We would like to read/write without maintaining the SA auth key or any sort of credentials .

Please guide what is missing here.

---

UPDATE : as per the suggestion by @boredabdel we checked and found that workload identity was already enabled on the GKE cluster as well as NodePool. We are using this module to create our cluster where it is already enabled by default. Still, we are facing connectivity issues.

Cluster Security:

NodePool Security:

Answer 1

This is for all who are looking for an answer to implement this solution via terraform. Please refer below:

Create a Kubernetes service account with annotation

resource "kubernetes_manifest" "service_account" {
  manifest = {
    "apiVersion" = "v1"
    "kind"       = "ServiceAccount"
    "metadata" = {
      "name"      = "KSA_NAME"
      "namespace" = "NAME"
      "annotations" = {
        "iam.gke.io/gcp-service-account" = "GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com"
      }
    }
    "automountServiceAccountToken" = true
  }
}

Create an IAM policy binding to allow the Kubernetes service account to act as the IAM service account.

resource "google_service_account_iam_binding" "service-account-iam" {
  service_account_id = "projects/PROJECT_ID/serviceAccounts/GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com"
  role               = "roles/iam.workloadIdentityUser"
  members            = [
    "serviceAccount:${var.project_id}.svc.id.goog[NAMESPACE/KSA-NAME]",
  ]
}

Add service account to the deployment/statefulset manifest like this

spec:
  serviceAccount: KSA-NAME
  containers:

NOTE :

1. GSA_PROJECT and PROJECT_ID are the same if you are using the same project for all objects.
2. K8s service account is created via resource "kubernetes_manifest" method because there is an open issue for it to create via resource "kubernetes_service_account" method where it is throwing an error stating. Waiting for default secret of "NAMESPACE/KSA_NAME" to appear

Answer 2

Seems like you are trying to use the Node Service Account to authenticate to GCS. You need to passe the Service Account Key to the app you are calling the API from as described in this doc

If you want Keyless authentication, my advice is to use Workload identity