Why does nginx return 404 on my Kubernetes Ingress route?

Question

I have two nginx web servers set up. One is set up on AWS Elastic Beanstalk, the other on Kubernetes using the stable/nginx-ingress helm chart.

The Elastic Beanstalk webserver forwards traffic from all subroutes of my domain to the Kubernetes nginx webserver. I can verify these are being forwarded correctly by checking the logs from the Kubernetes nginx. I use an Ingress resource to make sure this traffic is being forwarded to the right Kubernetes service.

Here is the problem : One of the two routes, the main / route, is forwarded to the correct service and returns a 200. The other route, /eks-test , is supposed to route to the same service, but returns a 404. How is this possible?

Specs :

The nginx on Kubernetes is running nginx 0.25.1 . I am running Kubernetes on AWS EKS version 1.14.

Nginx logs:

172.16.10.103 - [172.16.10.103] - - [12/Sep/2019:08:05:09 +0000] "GET / HTTP/1.0" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 703 0.004 [default-eks-test-repo-80] [] 172.16.10.100:8080 8 0.004 200 90dfa37364a5c43e57f12c5fb1a2d86f
172.16.40.108 - [172.16.40.108] - - [12/Sep/2019:08:05:12 +0000] "GET /eks-test HTTP/1.0" 404 9 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 730 0.002 [default-eks-test-repo2-80] [] 172.16.43.125:8080 9 0.004 404 ef1c81bba75dff2bdd2376799aa93c56

First nginx config (Elastic Beanstalk):

server {
    listen 80;

    server_name my.domain.com;

    location / {

        proxy_pass http://internal.my.domain.lan/;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Kubernetes Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: internal
  name: eks-test
  namespace: default
spec:
  rules:
    - host: my.domain.com
      http:
        paths:
          - path: /
            backend:
              serviceName: eks-test-repo
              servicePort: 80
          - path: /eks-test
            backend:
              serviceName: eks-test-repo
              servicePort: 80

Kubernetes Service:

kind: Service
apiVersion: v1
metadata:
  name: eks-test-repo
  namespace: default
  labels:
    name: eks-test-repo
spec
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: eks-test-repo
  type: ClusterIP

Nginx helm chart values (ones that are not default):

controller.ingressClass: internal

Answer 1

The answer to this question can be found here .

Basically, the paths were not being rewritten to / , but the webservers in the pods were only listening on the root path. Adding the nginx.ingress.kubernetes.io/rewrite-target: / annotation fixed the problem.

Answer 2

Please use the below configuration, It may resolve your issue.

annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: "/"
    ingress.kubernetes.io/configuration-snippet: |
      sub_filter_once on;
      sub_filter '<base href="/">' '<base href="/{{$definitionName}}/">';
spec:
  rules:
  - http:
      paths:
      - path: /{{$definitionName}}
        backend:
          serviceName: {{$definitionName}}
          servicePort: 80

Regards,

Vaibhav kharode

Answer 3

You should remove from Nginx config this line:

proxy_set_header        Host $host;

It rewrite host name, and Kubernetes can't find target host.