Nginx Ingress controller 502 bad gateway for large file uploads

Question

We recently upgraded from nginx ingress controller from 0.8.2 to 0.11.0, and started getting 502 bad gateway error on large file uploads around 10 MB or higher, we have set the client_max_body_size to 500m through proxy-body-size in the configmap and verified its set. The smaller files around 5-6 MB works fine.

There are no errors in the logs, just these messages.

redacted - [redacted] - - [25/Mar/2018:02:08:49 +0000] "POST /redacted/upload HTTP/1.1" 000 0 " https://redacted/ " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 3371263 10.850 [uploader-443] ----

and

[warn] 30684#30684: *42090 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000482, client: redacted, server: redacted, request: "POST /redacted/upload HTTP/1.1", host: "redacted", referrer: " https://redacted/ "

The proxied server is tomcat and requests do not make it to tomcat. We have tried increasing:

• timeouts
• proxy_buffers
• proxy_buffer_size

but nothing worked.

Going back to 0.8.2 version resolves the issue.

update 1: nginx.conf snippet

location /redacted/ {

port_in_redirect off;

set $proxy_upstream_name "redacted-443";

....

....

client_max_body_size "500m";

Answer 1

We recently upgraded from nginx ingress controller from 0.8.2 to 0.11.0, and started getting 502 bad gateway error on large file uploads around 10 MB or higher, we have set the client_max_body_size to 500m through proxy-body-size in the configmap and verified its set. The smaller files around 5-6 MB works fine.

There are no errors in the logs, just these messages.

redacted - [redacted] - - [25/Mar/2018:02:08:49 +0000] "POST /redacted/upload HTTP/1.1" 000 0 " https://redacted/ " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 3371263 10.850 [uploader-443] ----

and

[warn] 30684#30684: *42090 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000482, client: redacted, server: redacted, request: "POST /redacted/upload HTTP/1.1", host: "redacted", referrer: " https://redacted/ "

The proxied server is tomcat and requests do not make it to tomcat. We have tried increasing:

• timeouts
• proxy_buffers
• proxy_buffer_size

but nothing worked.

Going back to 0.8.2 version resolves the issue.

update 1: nginx.conf snippet

location /redacted/ {

port_in_redirect off;

set $proxy_upstream_name "redacted-443";

....

....

client_max_body_size "500m";

Answer 2

We recently upgraded from nginx ingress controller from 0.8.2 to 0.11.0, and started getting 502 bad gateway error on large file uploads around 10 MB or higher, we have set the client_max_body_size to 500m through proxy-body-size in the configmap and verified its set. The smaller files around 5-6 MB works fine.

There are no errors in the logs, just these messages.

redacted - [redacted] - - [25/Mar/2018:02:08:49 +0000] "POST /redacted/upload HTTP/1.1" 000 0 " https://redacted/ " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 3371263 10.850 [uploader-443] ----

and

[warn] 30684#30684: *42090 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000482, client: redacted, server: redacted, request: "POST /redacted/upload HTTP/1.1", host: "redacted", referrer: " https://redacted/ "

The proxied server is tomcat and requests do not make it to tomcat. We have tried increasing:

• timeouts
• proxy_buffers
• proxy_buffer_size

but nothing worked.

Going back to 0.8.2 version resolves the issue.

update 1: nginx.conf snippet

location /redacted/ {

port_in_redirect off;

set $proxy_upstream_name "redacted-443";

....

....

client_max_body_size "500m";

Answer 3

We recently upgraded from nginx ingress controller from 0.8.2 to 0.11.0, and started getting 502 bad gateway error on large file uploads around 10 MB or higher, we have set the client_max_body_size to 500m through proxy-body-size in the configmap and verified its set. The smaller files around 5-6 MB works fine.

There are no errors in the logs, just these messages.

redacted - [redacted] - - [25/Mar/2018:02:08:49 +0000] "POST /redacted/upload HTTP/1.1" 000 0 " https://redacted/ " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 3371263 10.850 [uploader-443] ----

and

[warn] 30684#30684: *42090 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000482, client: redacted, server: redacted, request: "POST /redacted/upload HTTP/1.1", host: "redacted", referrer: " https://redacted/ "

The proxied server is tomcat and requests do not make it to tomcat. We have tried increasing:

• timeouts
• proxy_buffers
• proxy_buffer_size

but nothing worked.

Going back to 0.8.2 version resolves the issue.

update 1: nginx.conf snippet

location /redacted/ {

port_in_redirect off;

set $proxy_upstream_name "redacted-443";

....

....

client_max_body_size "500m";

Answer 4

We recently upgraded from nginx ingress controller from 0.8.2 to 0.11.0, and started getting 502 bad gateway error on large file uploads around 10 MB or higher, we have set the client_max_body_size to 500m through proxy-body-size in the configmap and verified its set. The smaller files around 5-6 MB works fine.

There are no errors in the logs, just these messages.

redacted - [redacted] - - [25/Mar/2018:02:08:49 +0000] "POST /redacted/upload HTTP/1.1" 000 0 " https://redacted/ " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" 3371263 10.850 [uploader-443] ----

and

[warn] 30684#30684: *42090 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000482, client: redacted, server: redacted, request: "POST /redacted/upload HTTP/1.1", host: "redacted", referrer: " https://redacted/ "

The proxied server is tomcat and requests do not make it to tomcat. We have tried increasing:

• timeouts
• proxy_buffers
• proxy_buffer_size

but nothing worked.

Going back to 0.8.2 version resolves the issue.

update 1: nginx.conf snippet

location /redacted/ {

port_in_redirect off;

set $proxy_upstream_name "redacted-443";

....

....

client_max_body_size "500m";

Answer 5

I got the same issue

First I was getting 413 which was solved by adding

nginx.ingress.kubernetes.io/proxy-body-size: "10m"

Later on the service started giving 502 . When looked into the pod , the pod was OOMKilled when the file was uploaded. So we need increase the resource limits of our containers.

 containers:
    - name: test-service
      image: test-service:develop
      imagePullPolicy: Always
      resources:
        requests:
          memory: 256Mi
        limits:
          memory: 512Mi