Address null for ingress in EKS with classic LB

Question

when I create ingress it is created with no address and when I describe my ingress I see message

Failed build model due to WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
  status code: 403, request id: 5423ee08-9a72-47fe-8389-3f50ce78b0e5

and when I check pod logs for aws loadbalancer controller, see the similar error

{"level":"error","ts":1674658664.611337,"logger":"controller-runtime.manager.controller.ingress","msg":"Reconciler error","name":"catch","namespace":"sa-backup","error":"WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: b4a791b1-f56b-4d4b-84b4-a7b6bc5ff8b9"}

I can confirm that classic load balancer is created fine and ingressRoutes are working. Just problem with ingress controller

Answer 1

Your AWS Load Balancer Controller needs access to the AWS API. The standard way to give API access to a pod in EKS is using IRSA ( https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html ) which allows the Pod to assume roles through OIDC in the cluster and a trust relationship with the AWS API. This trust relationship needs setup before IRSA will work ( https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-enable-IAM.html ).

The AWS IAM Role that your AWS LB Controller is trying to assume will need to have a Trust policy that allows the OIDC endpoint of your EKS cluster that references the Namespace and ServiceAccount name used by the pod.

Assuming you are in us-east-2 and the service account is in the aws-lbc namespace and named aws-load-balancer-controller , the ServiceAccount that the pod is using will need an annotation specifying which AWS IAM role to use:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam:<account number>:role/<iam role name>
  name: aws-load-balancer-controller
  namespace: aws-lbc

The trust policy on the AWS IAM role should look something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account number>:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/<OIDC endpoint ID>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-2.amazonaws.com/id/<OIDC endpoint ID>:sub": "system:serviceaccount:aws-lbc:aws-load-balancer-controller"
                }
            }
        }
    ]
}

The <OIDC endpoint ID> can be retrieved from the AWS console under your EKS cluster Overview -> Details -> OpenID Connect provider URL (it's the 32 character string after /id ).