Random/Intermittent 502 gateway errors with nginx and node deployments using proxy_pass on a k8s cluster

Question

My current configuration is as follows

Bare metal cluster. SELINUX status is off.

1. A nginx deployment
2. A nodejs deployment

I am serving static content through the ngnix service and the dynamic content using the node service. Below is my nginx configuration.

worker_processes  4;

#error_log  logs/error.log  info;
error_log  /dev/stdout  info;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    tcp_nopush      on;

    #keepalive_timeout  0;
    #keepalive_timeout  5;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /dev/stdout  main;


    server {
        listen       1080;
        server_name  localhost $hostname;

        root  /usr/share/nginx/static/;
    
        # static content
        location ~ some-regex {
            alias /usr/share/nginx/static/;
            
            # handle cors see 'NGINX-Cookbook' for production quality
            add_header 'Access-Control-Allow-Origin' '*';
        }

        # forward request to node-service
        location / {
             client_max_body_size 128M;
             proxy_buffer_size 256k;
             proxy_buffers 4 512k;
             proxy_busy_buffers_size 512k;
             proxy_http_version 1.1;
            #  proxy_set_header Connection "";
            #  proxy_http_version 1.1;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection keep-alive;
             proxy_set_header Host $http_host;
             proxy_cache_bypass $http_upgrade;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            #  proxy_socket_keepalive on;
             proxy_pass http://nodeserver:3000;
        }
    }

    include servers/*;
}

In my ingress I am hitting the nginx service. I am able to forward my requests correctly to node service few times and get the proper response but around 50% of the time the request fails with a 502 bad gateway error and I see this error in the nginx pod logs

[error] 20#20: *187 connect() failed (111: Connection refused) while connecting to upstream, client: 10.44.0.2, server: localhost, request: "GET /path HTTP/1.1", upstream: "http://node-service-clusterip:3000/path", host: "my-nginx-node.example.com"

I have tried multiple directives from the nginx documentation but to no avail. Any help would be much appreciated

Answer 1

There was a mistake in my kubernetes label selectors. I was using same selectors for multiple deployments which caused issues in the routing.