Is this possible to connect kubernetes ingress ALB with terraform managed alb/route 53 while serving requests on same dns name?

Question

I have a setup in AWS with different lambdas - all managed by terraform. Now only requests to path like https://example.com/home or https://example.com/blog are forwarded to different AWS lambdas using route53 record and ALB with different rules - here is an example for /home/ path:

resource "aws_route53_record" "dns-record" {
  name    = "example.com"
  zone_id = var.zone_id
  type    = "CNAME"
  ttl     = "300"
  records = [aws_lb.alb.dns_name]
}

resource "aws_lb" "alb" {
  name               = "my-alb..."
........
}

resource "aws_lb_listener" "alb-in-443" {
  load_balancer_arn = aws_lb.alb.arn
  port              = "443"
  protocol          = "HTTPS"
........
  default_action {
    type = "fixed-response"

    fixed_response {
      content_type = "text/plain"
      message_body = "Fixed response content"
      status_code  = "200"
    }
}

resource "aws_lb_listener_rule" "home-in-443" {
  listener_arn = aws_lb_listener.alb-in-443.arn
  priority     = 100

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.home-alb-tg.arn
  }

  condition {
    path_pattern {
      values = ["/home/*"]
    }
  }
}

resource "aws_lb_target_group" "home-alb-tg" {
  name        = "home-alb-tg-lambda"
  target_type = "lambda"
  vpc_id      = data.aws_vpc.vpc.id
}

resource "aws_lambda_permission" "home-lb-lambda-permission" {
......
}

resource "aws_lb_target_group_attachment" "home-alb-tg-attachment" {
.....
}

So far all works fine, but now I need to add AWS EKS cluster and forward all requests to https://example.com to EKS - while continuing to serve /home or /blog with AWS lambda. I can create another ALB with AWS Load balancer controller and then forward requests using such Ingress resource in front of my service, with such config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress-service
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
  labels:
    app: my-app
spec:
  rules:
  - host: "example.com"
    http:
      paths:
        - path: /*
          backend:
            serviceName: my-service
            servicePort: 80

But this ALB will be detached from route53 and furthermore, such path will conflict with path defined in terraform loadbalancer rule described above. On the other hand, I can define all conditions for all paths (/home,/blog,etc) in ingress config above - but I won't be able to bind them with lambdas.

So, question is - is such setup with serving main url from EKS and different paths with lambdas even possible? Maybe this can be done with aws cloudfront somehow?

Answer 1

Well, it seems that this is technically possible with Cloudfront - here is config that I used. I created 2 different origins -one points to dns name from k8s ALB and another points to dns name from ALB created with terraform. Here is config:

data "aws_lb" "eks-lb" {
  name = "k8s-default-appservi-3f93453"  -- we need to get alb name created in k8s - this doesn't look good but we can't specify alb name right now
}

resource "aws_cloudfront_distribution" "my-distribution" {

  enabled         = true
  is_ipv6_enabled = true
  aliases = "example.com"

  origin {
    domain_name = data.aws_lb.eks-lb.dns_name  - use DNS name from eks alb here
    origin_id   = "my-app"

    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  origin {
    domain_name = aws_lb.alb.dns_name   - use DNS name from "alb" lb created in terraform above 
    origin_id   = "home"

    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  default_cache_behavior {
    allowed_methods  = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "my-app"

    forwarded_values {
      headers = [ "Host" , "Origin"]
      query_string = true
      cookies {
        forward = "all"
      }
    }

    min_ttl                = 0
    default_ttl            = 0
    max_ttl                = 0
    viewer_protocol_policy = "redirect-to-https"
  }

  ordered_cache_behavior {
    path_pattern     = "/home/*"
    allowed_methods  = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "home"

    forwarded_values {
      headers = [ "Host", "Origin" ]
      query_string = true
      cookies {
        forward = "all"
      }
    }

    min_ttl                = 0
    default_ttl            = 0
    max_ttl                = 0
    viewer_protocol_policy = "redirect-to-https"
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn = "cert.arn"
    ssl_support_method  = "sni-only"
  }
}

But I don't like this solution because k8s ALB dns name should be hardcoded and also we have aws resources (ALB and target group) which are not managed by Terraform and which stayed in account even after I deleted both aws load balancer controller and ingress service ( github issue ). So maybe better solution will be to change AWS load balancer controller to just ingress-nginx with NLB before it and use external-dns to create dns record which will be used in cloudfront configuration.