gRPC in EKS on AWS with Nginx-Ingress-Controller

Question

I have a gRPC server set up in AWS EKS, and use Nginx-Ingress-Controller to perform load balancing. I try to terminate TLS at NLB, by setting the gRPC server ingress to like

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: my-grpc
  namespace: myspace
spec:
  rules:
    - host: my.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: grpc-server
              servicePort: 8080

Also, I use Amazon Certificate Manager to manage the TLS for NLB, so I have to change the Helm Chart of Nginx-Ingress-Controller Value.yaml the following fields

controller:
  service:
    enabled: true
    annotations: 
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxxxxxxx
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443,8443"
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

    targetPorts:
      http: http
      https: http

The problem is, I could not successfully call through 443 port and make the client connect to the gRPC server.

The problem happens between NLB and Nginx, but what and why is unknown. Any kind of help will be appreciated.

Note: I am aware of the example ingress-nginx has TLS field, but what I should put here if I am using ACM.

Answer 1

I finally got something working through AWS + NLB + EKS. I started with this basic grpc app

Swap out ever instance of 'fortune-teller.stack.build' with your URL (eg example.com)

In the ingress, I had to change tls -> secret Name to the tls-secret that I created earlierusing this guide

kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: fortune-ingress
  namespace: default
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: fortune-teller-service
          servicePort: grpc
  tls:
  - secretName: tls-secret
    hosts:
      - example.com

In cert.yaml, change

1. spec -> Domain to my URL (example.com)

In svc.yaml, added annotations and "type: LoadBalancer"

apiVersion: v1
kind: Service
metadata:
  name: fortune-teller-service
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-internal: "false"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:<REGION>:<MY ACCOUNT>:certificate/<CERT ID>"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
spec:
  type: LoadBalancer
  selector:
    k8s-app: fortune-teller-app
  ports:
  - port: 50051
    targetPort: 50051
    name: grpc

Once all of that is deployed via the Readme, find the NLB in the AWS console.

1. Listeners tab -> select the listener (50051) and click 'edit'
2. Change protocol to TLS and I changed the port to 443
3. Under ALPN Policy, Select "HTTP2Only"
4. Keep the same target group under "Default Actions -> forward to"
5. Select a policy version ( I chose ELBSecurityPolicy2016-08 but will be upgrading this) and your cert from ACM that is tied to your URL
6. Click "Update"
7. Point your DNS name to your NLB

Now wait...as the listener update seems to take a few minutes.

grpcurl example.com:443 build.stack.fortune.FortuneTeller/Predict

I was also able to get my app working successfully by swapping out image, port and of course the names.

Mistakes I was making that made it not work for me:

• Not having a TLS secret in the ingress
• Trusting the AWS docs when they say that your Target group has to be a "TLS Target Group". If you follow this example, you never swap the TG. The TG stays as a TCP TG and still works.

Answer 2

Thank CrookedSmile for his answer, but it turns out my problem has a root cause in my Nginx-Ingress-Controller Setup and I finally solve it in this way:

I mistakenly think to terminate TLS at NLB and Terminate TLS at Nginx is the same thing when I use Nginx-Ingress-Controller. But it is not.

In my situation, I do not need to terminate TLS at NLB, I need to terminate it at Nginx.

So I do the following 2 things:

1. I revert what I do following this section AWS L7 ELB SSL Termination .

2. Nginx is configured by my application Ingress file. So I must set the TLS fields of the gRPC server ingress like ingress.yaml example here .

Then TLS is successfully terminated at Nginx and I could call my gRPC server behind NLB with Nginx-Ingress-Controller in between.

Additional Note: Currently, Nginx-Ingress-Controller does not support ACM yet (July 1 2020), so you have to manage each certificate on your own.

Answer 3

@CrookedSmile - When you deploy a service with type as LoadBalancer, it creates a Network Loadbalancer in your AWS account. Now, when you create an Nginx Ingress controller, that's another Loadbalancer that is being created under the hood. As per your instructions, you are exposing the service over Loadbalancer - created as a part of the service type. When you modify the target group lister port, it breaks the communication between your Ingress and this service Loadbalancer because as per your Kubernetes service manifests, desired states of the Loadbalancer listener is 50051.

You can follow the order below to get the sample grpc app deployed.

https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc

1. Deploy the fortune-teller app

$ kubectl create -f app.yaml

2. Deploy the service as type: ClusterIP

 $ kubectl create -f svc.yaml

3. Deploy the nginx-ingress controller

 $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/do/deploy.yaml

4. Deploy the ingress resources

$ kubectl create -f ingress.yaml

4. Finally, add a host entry to spoof your domain name For eg: "fortune-teller.stack.build", with the A record of your Ingress LB.

5. Validation

$ grpcurl --insecure fortune-teller.stack.build:443 build.stack.fortune.FortuneTeller/Predict { "message": "Fremen add life to spice!" }

In a nutshell, here is the workflow.

GRPC Request --> Nginx Ingress NLB – > Backend Service (Cluster IP) --> Application pods