ESP on GKE fails if the service account key JSON file is omitted

Question

I'm running ESPv2 on GKE. The container starts only if the service account key JSON file is provided using the --service_account_key flag. However this flag is documented under Non GCP Platform Deployment . Furthermore, proxy startup options states

The only time you need to specify this option is when ESP is running on platforms other than Google Cloud [...]

Therefore I don't think --service_account_key is necessary when running containers in GKE.

Because I'm using Workload Identity the GKE workload runs using a kube.netes service account which impersonates a Google service account. The steps to achieve this are detailed in Configure applications to use Workload Identity . However, I'm using Terraform so don't perform these steps myself, rather I rely on Terraform's workload-identity submodule which is configured to

• Use an existing Google service account ( use_existing_gcp_sa = true )
• Create a new Kube.netes service account ( use_existing_k8s_sa = false )
• Annotate the kube.netes service account ( annotate_k8s_sa = true )

Terraform's workload-identity submodule also takes care of creating the Google service account binding to roles/iam.workloadIdentityUser . Therefore all the steps in Configure applications to use Workload Identity are implemented by Terraform.

The service account key JSON file that I'm passing to the ESP is the Google Service account (which acts on behalf of the kube.netes service account) but it is not the default compute service account . Therefore, intuitively, it makes sense to me that ESP needs this. Yet all the documentation states it should not be needed if running on GCP.

Has anyone got ESP running on GKE using workload identity and not using the default compute service account?

My Kube.netes deployment config with some ESP flags removed for clarity

---
apiVersion: apps/v1
kind: Deployment
spec:
  spec:
    # Kubernetes service account in default namespace.
    serviceAccountName: my-k8s-sa
    volumes:
      - name: account-key
        secret:
          secretName: account-key-secret
    containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:2
        args: [
            # Google service account (should not be needed in GCP)
            "--service_account_key=/etc/nginx/creds/credentials.json",
          ]
        volumeMounts:
          - mountPath: /etc/nginx/creds
            name: account-key
            readOnly: true

Answer 1

As you mention, ESPv2 needs access tokens. When you deploy ESP or ESPv2 on Google Cloud environments, such as GKE or Compute Engine, or ESP and ESPv2 these obtain access tokens for you through the Google Cloud metadata service , so due this ESP will fetch and use credentials from Compute engine instance in order to work on the GCP environment.

I recommend review the service account used running this command inside your pod to validate the service account:

curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/

If the service accounts are correctly configured, the IAM service account email address is listed as the active.

Also, note that implementing workload identity has some limitations and could be some Time Out errors at Pod start up, but you can resolve the issue following this troubleshooting guide . If after this troubleshooting, you are still facing issues, I will suggest collect the startup logs from the ESPv2 and reach to GCP support to shared them in this link due to the fact that it could be personal (sensitive) information that I think could be shared and needs a better chanel to keep your information confidential.