nginx Ingress controller logs 499

Question

I have an azure Kubernetes cluster, deployed as shown below. I have a long-running api which takes about 2 minutes to complete. Sometimes when the request is being processed on the kubernetes POD, nginx ingress controller logs an error with status code 499. As per nginx log, 499 is client closed connection. I have set the nginx time out to 10 minutes (for test). I want to understand who is the client for nginx here. Is it the application load balancer or the application POD? Please note that the underlying .NET core Api continues with the request processing

Log:

[10.XX.XX.5] "POST /test HTTP/1.1" 499 0 " https://test.test/ " 2345 120.601 [default-test-service-80] 10.xx.xx.xx:80 0 120.603

Answer 1

To learn about what does the log mean, you need to know how does it define the logs format for the Nginx. So you can go inside the pod of the Nginx ingress and look at the config for the Nginx, and it shows like this:

log_format upstreaminfo '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';

And you can find the document about how does the Nginx defines the variables in the logs format. The document describes here:

$remote_addr - x.x.x.x - The IP address of the client making the request.

And in your exact log, the remote address is "10.XX.XX.5". It's the IP address of the AKS node which hosts the Nginx ingress pod.

A. The client for the Nginx here is the AKS node that hosts the Ningx Ingress pod.

Finally, as I think, the requests routing path is AGW -> ALB -> AKS_node -> Nginx_ingress_controller -> service -> application_pod.

Answer 2

Check your application gateway timeout

HTTP Settings > your setting name > Request time-out (seconds)