Terraform cycle with AWS and Kubernetes provider

Question

My Terraform code describes some AWS infrastructure to build a Kubernetes cluster including some deployments into the cluster. When I try to destroy the infrastructure using terraform plan -destroy I get a cycle:

module.eks_control_plane.aws_eks_cluster.this[0] (destroy)
module.eks_control_plane.output.cluster
provider.kubernetes
module.aws_auth.kubernetes_config_map.this[0] (destroy)
data.aws_eks_cluster_auth.this[0] (destroy)

Destroying the infrastructure works by hand using just terraform destroy works fine. Unfortunately, Terraform Cloud uses terraform plan -destroy to plan the destructuion first, which causes this to fail. Here is the relevant code:

excerpt from eks_control_plane module:

resource "aws_eks_cluster" "this" {
  count = var.enabled ? 1 : 0

  name     = var.cluster_name
  role_arn = aws_iam_role.control_plane[0].arn
  version  = var.k8s_version

  # https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
  enabled_cluster_log_types = var.control_plane_log_enabled ? var.control_plane_log_types : []

  vpc_config {
    security_group_ids = [aws_security_group.control_plane[0].id]
    subnet_ids         = [for subnet in var.control_plane_subnets : subnet.id]
  }

  tags = merge(var.tags,
    {
    }
  )

  depends_on = [
    var.dependencies,
    aws_security_group.node,
    aws_iam_role_policy_attachment.control_plane_cluster_policy,
    aws_iam_role_policy_attachment.control_plane_service_policy,
    aws_iam_role_policy.eks_cluster_ingress_loadbalancer_creation,
  ]
}

output "cluster" {
  value = length(aws_eks_cluster.this) > 0 ? aws_eks_cluster.this[0] : null
}

aws-auth Kubernetes config map from aws_auth module:

resource "kubernetes_config_map" "this" {
  count = var.enabled ? 1 : 0

  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }
  data = {
    mapRoles = jsonencode(
      concat(
        [
          {
            rolearn  = var.node_iam_role.arn
            username = "system:node:{{EC2PrivateDNSName}}"
            groups = [
              "system:bootstrappers",
              "system:nodes",
            ]
          }
        ],
        var.map_roles
      )
    )
  }

  depends_on = [
    var.dependencies,
  ]
}

Kubernetes provider from root module:

data "aws_eks_cluster_auth" "this" {
  count = module.eks_control_plane.cluster != null ? 1 : 0
  name  = module.eks_control_plane.cluster.name
}

provider "kubernetes" {
  version = "~> 1.10"

  load_config_file       = false
  host                   = module.eks_control_plane.cluster != null ? module.eks_control_plane.cluster.endpoint : null
  cluster_ca_certificate = module.eks_control_plane.cluster != null ? base64decode(module.eks_control_plane.cluster.certificate_authority[0].data) : null
  token                  = length(data.aws_eks_cluster_auth.this) > 0 ? data.aws_eks_cluster_auth.this[0].token : null
}

And this is how the modules are called:

module "eks_control_plane" {
  source  = "app.terraform.io/SDA-SE/eks-control-plane/aws"
  version = "0.0.1"
  enabled = local.k8s_enabled

  cluster_name          = var.name
  control_plane_subnets = module.vpc.private_subnets
  k8s_version           = var.k8s_version
  node_subnets          = module.vpc.private_subnets
  tags                  = var.tags
  vpc                   = module.vpc.vpc

  dependencies = concat(var.dependencies, [
    # Ensure that VPC including all security group rules, network ACL rules,
    # routing table entries, etc. is fully created
    module.vpc,
  ])
}


# aws-auth config map module. Creating this config map will allow nodes and
# Other users to join the cluster.
# CNI and CSI plugins must be set up before creating this config map.
# Enable or disable this via `aws_auth_enabled` variable.
# TODO: Add Developer and other roles.
module "aws_auth" {
  source  = "app.terraform.io/SDA-SE/aws-auth/kubernetes"
  version = "0.0.0"
  enabled = local.aws_auth_enabled

  node_iam_role = module.eks_control_plane.node_iam_role
  map_roles = [
    {
      rolearn  = "arn:aws:iam::${var.aws_account_id}:role/Administrator"
      username = "admin"
      groups = [
        "system:masters",
      ]
    },
    {
      rolearn  = "arn:aws:iam::${var.aws_account_id}:role/Terraform"
      username = "terraform"
      groups = [
        "system:masters",
      ]
    }
  ]
}

Removing the aws_auth config map, which means not using the Kubernetes provider at all, breaks the cycle. The problem is obviously that Terraform tries to destroys the Kubernetes cluster, which is required for the Kubernetes provider. Manually removing the resources step by step using multiple terraform apply steps works fine, too.

Is there a way that I can tell Terraform first to destroy all Kubernetes resources so that the Provider is not required anymore, then destroy the EKS cluster?

Answer 1

You can control the order of destruction using the depends_on meta-argument, like you did with some of your Terraform code.

If you add the depends_on argument to all of the required resources that are needing to be destroyed first and have it depend on the eks-cluster Terraform will destroy those resources before the cluster.

You can also visualize your configuration and dependency with the terraform graph command to help you make decisions on what dependencies need to be created.

https://www.terraform.io/docs/cli/commands/graph.html