Ephemeral environments using GCP, Cloud Build and Terraform workspaces

Question

As the title says, trying to leverage Cloud Build and Terraform workspaces to create arbitrary ephemeral environments in the same GCP project, mapping to a branch.

My Cloud Build pipeline works but terraform apply doesn't seem to be honouring the workspace I'm trying to set, in that resources from the previous builds are destroyed, even if they were supposed to be in another workspace.

cloud_build.yaml

  - id: 'tf workspace'
    name: 'hashicorp/terraform:1.0.2'
    args: ['workspace','new',$BRANCH_NAME]

  - id: 'tf workspace'
    name: 'hashicorp/terraform:1.0.2'
    args: ['workspace','select',$BRANCH_NAME]

  - id: 'tf init'
    name: 'hashicorp/terraform:1.0.2'
    args: ['init']

  - id: 'tf plan'
    name: 'hashicorp/terraform:1.0.2'
    args: ['plan',"-var","branch_name=$BRANCH_NAME","-var","project_id=$PROJECT_ID"]

  - id: 'tf apply'
    name: 'hashicorp/terraform:1.0.0'
    args: ['apply',"-auto-approve","-var","branch_name=$BRANCH_NAME","-var","project_id=$PROJECT_ID"]

The backend is simply:

terraform {
  backend "gcs" {
    bucket = "my-tfstate-bucket"
  }
}

Am I missing something about workspaces? It would seem they're not really being reflected in the remote state. I know you're able to specify a specific workspace in the tf backend config, but since these workspaces are dynamic, I'd been hoping that setting the workspace via the CLI would set some kind of namespace in the uploaded remote state and take care of the separation.

EDIT Just to confirm that I am naming all of my components differently, with the current branch name as a suffix, but TF will still destroy components created when a different branch was run.

Answer 1

From the best of my understanding, terraform workspaces allow you to store Terraform state in multiple, separate, named (state) files.

Thus workspaces are only about state files (naming). They don't affect GCP resource naming.

I understand that you would like to deploy GCP resources from different git branches into one GCP project.

In majority of cases, GCP project is a "resource naming scope" - it means that one cannot create more than one resource (of a given type) with the chosen name in one GCP project. For example, I cannot create 2 pubsub topic with identical names, or create 2 cloud functions with identical names, etc. in one GCP project.

Under assumption that inside your terraform files (this is my guess, as there are no terraform files in the original question), you don't handle dependency - GCP resource name as a function of a git branch (for different branches the deployed resources should have different names) - I guess that a new workspace deployment ( apply ) compares what already exist in a project with what is described in TF files, and with is recorded in thee state file (empty for a new workspace), and makes a decision to recreate all resources and record that in the new workspace state file. I think that the old workspace state file is kept as is (untouched, unmodified).

To resolve that issue, it may be necessary to develop some mechanism (inside TF files), so that every GCP resource, described in those files, gets the name (which must be unique in the GCP project) according to (functionally dependent on) the git branch name, so that there is no interference between resources deployed from different git branches.

Answer 2

Figured it out. Because cloud build caches the Terraform workspace between builds, init needs to be called before the workspace commands. Switching the order of my steps solved the problem, I now see multiple state files for each workspace name in my bucket.