Terraform GCS backend writing .tflock failed. 403 access denied

Question

I am trying to use Terraform with a Google Cloud Storage backend, but I'm facing some issues when executing this in my CI pipeline.

I have set the GOOGLE_APPLICATION_CREDENTIALS to my service account JSON keyfile, but whenever I try to init Terraform, I get the following errors:

Error loading state: 2 errors occurred:
    * writing "gs://[my bucket name]/state/default.tflock" failed: googleapi: Error 403: Access denied., forbidden
    * storage: object doesn't exist

I have tried all documented methods of authentication, but still no luck.

Answer 1

Turns out only the second error was actually relevant and there were no authentication issues after all.

My remote backend only contained my custom workspace state files and no default state. Since terraform init needs to be executed before being able to switch to a workspace, it was looking for a default.tflock / default.tfstate file that did not exist.

From my local workstation I initialized the default workspace, which created the file that Terraform was looking for.

I wasted a good few hours trying to debug a service account authentication issue that did not exist. I hope this answer can save someone else from that rabbit hole...