Terraform config isn't using output from other file for already created resource, instead tries to recreate it and fails (security group id)

Question

In terraform/aws/global/vpc/security_groups.tf I have the below code to create my bastion security group, and the output.tf file as well which is below. But in terraform/aws/layers/bastion/main.tf (code also below) I reference that security group as I need its security group ID to create my EC2 instance, the issue I have is that rather than getting the ID from the already existing security group created by the /vpc/security_groups.tf config it tries to create the whole security group and the run obviously fails because it already exists. How can I change my code to get the ID of the existing SG? I don't want to create my SG in the same config file as my instance, some of my security groups are shared between different resources. I am using Terraform Cloud and VPC has its own workspace, so I assume this could actually be an issue with the states being different.. is there a work around for this?

terraform/aws/global/vpc/security_groups.tf

    provider "aws" {
    region = "eu-west-1"
}

resource "aws_security_group" "bastion" {
  name        = "Bastion_Terraform"
  description = "Bastion SSH access Terraform"
  vpc_id      = "vpc-12345"

  ingress {
    description = "Bastion SSH"
    from_port   = ##
    to_port     = ##
    protocol    = "##"
    cidr_blocks = ["1.2.3.4/56"]
  }
  ingress {
    description = "Bastion SSH"
    from_port   = ##
    to_port     = ##
    protocol    = "##"
    cidr_blocks = ["1.2.3.4/0"]    
  }
  egress {
    description = "Access to "
    from_port   = ##
    to_port     = ##
    protocol    = "tcp"
    security_groups = ["sg-12345"]
  }
  egress {
    description = "Access to ##"
    from_port   = ##
    to_port     = ##
    protocol    = "tcp"
    security_groups = ["sg-12345"]
  }

  tags = {
    Name = "Bastion Terraform"
  }
}

terraform/aws/global/vpc/outputs.tf

output "bastion-sg" {
  value = aws_security_group.bastion.id
}

terraform/aws/layers/bastion/main.tf

    provider "aws" {
    region = var.region
}

module "vpc" {
    source = "../../global/vpc"
}

module "ec2-instance" {
    source = "terraform-aws-modules/ec2-instance/aws"

    name = "bastion"
    instance_count = 1
    ami = var.image_id
    instance_type = var.instance_type
    vpc_security_group_ids = ["${module.vpc.bastion-sg}"]
    subnet_id = var.subnet
    iam_instance_profile = var.iam_role

    tags = {
        Layer = "Bastion"
    }
}

Answer 1

When you have a child module block like this in a TF module:

module "ec2-instance" {
    source = "terraform-aws-modules/ec2-instance/aws"

    name = "bastion"
    instance_count = 1
    ami = var.image_id
    instance_type = var.instance_type
    vpc_security_group_ids = ["${module.vpc.bastion-sg}"]
    subnet_id = var.subnet
    iam_instance_profile = var.iam_role

    tags = {
        Layer = "Bastion"
    }
}

It doesn't just reference that child module, it instatiates a completely new instance of it unique only to the parent module and its state. Think of this not like an assignment or a pointer but the construction of a whole new instance of the module (using the module as a template) with all of its resources created again.

You will need to either directly reference the outputs of the child module in the parent module that has its module block or you will need to use a terraform_remote_state data source or Terragrunt dependency to load the outputs from the state file.