GCP - Cloud Composer 2 - Create operation on this environment failed

Question

I am trying to create a default Composer 2 Instance on GCP and get the Errors:

CREATE operation on this environment failed 32 minutes ago with the following error message:
Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING
description: "No agent response published."
...

or

CREATE operation on this environment failed 32 minutes ago with the following error message:
Environment couldn't be created, but no error was surfaced. This can be cause by a lack of 
proper permissions. Check if this environment's service account ... .iam.gserviceaccount.com 
has the 'roles/composer.worker' role and there is no firewall inhibiting internal 
communications set.

I already tried to add the Composer Worker role to the service account and all other required roles (eg Cloud Composer v2 API Service Agent Extension) in https://cloud.google.com/composer/docs/composer-2/access-control (for public as well as for private, eventhough instance is public).

I looked into the GKE instance and found the Pod composer-agent failing with:

Traceback (most recent call last): File "composer_agent.py", line 467, in <module> main() File "composer_agent.py", line 292, in main responses = pubsub_subscriber.pull() (...)
oauth2client.client.HttpAccessTokenRefreshError: Failed to retrieve
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/(...)
compute@developer.gserviceaccount.com/token from the Google Compute Enginemetadata service. 
Response:
{'date': 'Thu, 17 Feb 2022 10:29:46 GMT', 'status': '403', 'content-length': '668', 'content- 
type': 'text/plain; charset=utf-8', 'x-content-type-options': 'nosniff'}

So I assume there is still some permission issue but I can not figure out what, Composer 1 Instances can be created without a problem, as well for a different project Composer 2 Instances with the same permissions on the service accounts.

I also tried to create different than default compute service account with the required permissions but also without success. I also checked that the service account I am adding permissions to is actually the service account sending the request in the composer-agent and is sending the environment creation request to the GKE cluster.

I hope anyone can help, who faced similar issues or knows more about the error occuring in composer-agent, thank you very much!

Answer 1

After being in contact with the Google Support Team, the solution was to manually enable the "IAM Service Account Credentials API". There was no issue in Service Account Rights or Firewall settings.