您如何正确设置 Terraform 以使用跨账户 IAM 承担角色身份验证来管理 AWS EKS 集群？

Question

We're in the process of setting up some green field AWS infrastructure.

We have at the organisation account level an IAM user that Terraform authenticates as using access keys. We've then setup our Terraform code to "assume role" into our sub accounts within their respective git repos (we have 1 git repo per account). Something like,

provider "aws" {
  assume_role {
    role_arn = "arn:aws:iam::XXXXXXXXXX:role/TerraformCloudRole"
  }
}

We're running into issues setting up an EKS cluster using the terraform-aws-modules/eks/aws module. The cluster creates fine, but we've set manage_aws_auth_configmap = true so we can provide IAM roles/users and manage what they can authenticate against. We're actually seeing multiple errors depending on where we do creates or updates, and some subtle changes to the code. Essentially they are,

Error: The configmap "aws-auth" does not exist
with module.eks_main.module.eks.kubernetes_config_map_v1_data.aws_auth[0]
on .terraform/modules/eks_main.eks/main.tf line 470, in resource "kubernetes_config_map_v1_data" "aws_auth":

Or

Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: connect: connection refused
with module.eks_main.module.eks.kubernetes_config_map_v1_data.aws_auth[0]
on .terraform/modules/eks_main.eks/main.tf line 470, in resource "kubernetes_config_map_v1_data" "aws_auth":

We did some Googling and we found this issue . We added a provider and this seemed to solve some issues, specifically using this approach . The reason was because the exec route didn't work for us. It appeared to be trying to execute the AWS CLI command using the base access keys and not the assumed role. But the errors are back when we're making updates to the cluster or trying to run a destroy. It doesn't appear to be picking up the provider for some reason, The latter error above is during the plan phase. not apply.

So to my question . How do we setup Terraform to connect to/manage an EKS cluster properly when AWS assume-role/cross-account is involed?

Answer 1

maybe help:

provider "aws" {
  alias  = "main"
  region = var.aws_region
}

data "aws_eks_cluster" "selected" {
  provider = aws.main
  name     = local.eks_cluster_name
}

data "aws_eks_cluster_auth" "selected" {
  provider = aws.main
  name     = local.eks_cluster_name
}

provider "kubernetes" {
  host                   = element(concat(data.aws_eks_cluster.selected[*].endpoint, tolist([""])), 0)
  cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.selected[*].certificate_authority.0.data, tolist([""])), 0))
  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    args        = ["token", "-i", element(concat(data.aws_eks_cluster.selected[*].id, tolist([""])), 0)]
    command     = "aws-iam-authenticator"
  }
}

provider "helm" {
  kubernetes {
    host                   = element(concat(data.aws_eks_cluster.selected[*].endpoint, tolist([""])), 0)
    cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.selected[*].certificate_authority.0.data, tolist([""])), 0))
    exec {
      api_version = "client.authentication.k8s.io/v1alpha1"
      args        = ["token", "-i", element(concat(data.aws_eks_cluster.selected[*].id, tolist([""])), 0)]
      command     = "aws-iam-authenticator"
    }
  }
}

I'm not sure this will up to date, but I believe this should help you a lot, you can change the command to tho, as it use aws-iam-authenticator binary