Kubernetes Nginx Ingress 文件上傳返回 502

Question

我正在嘗試通過 nginx 入口從客戶端上傳文件。 在收到 413 響應后，我在入口設置了以下注釋；

Annotations:   nginx.ingress.kubernetes.io/body-size: 1024m
               nginx.ingress.kubernetes.io/client-body-buffer-size: 50m
               nginx.ingress.kubernetes.io/client-max-body-size: 50m
               nginx.ingress.kubernetes.io/proxy-body-size: 1024m
               nginx.ingress.kubernetes.io/proxy-buffer-size: 32k
               nginx.ingress.kubernetes.io/proxy-buffers-number: 8

客戶端是 Angular 應用程序。 它在請求正文中發送文件的 base64 字符串。 我已經嘗試上傳幾 KB 的圖像，所以我絕對沒有達到任何這些限制。 我是 Kubernetes 的新手。 我是否需要重新啟動入口才能使這些注釋生效？

我也嘗試過創建一個 ConfigMap；

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-ingress-configuration
  namespace: development
  labels:
    app.kubernetes.io/name: [name of ingress]
    app.kubernetes.io/part-of: [name of ingress]
data:
  proxy-connect-timeout: "50"
  proxy-read-timeout: "120"
  proxy-send-timeout: "120"
  body-size: "1024m"
  client-body-buffer-size: "50m"
  client-max-body-size: "50m"
  proxy-body-size: "1024m"
  proxy-buffers: "8 32k"
  proxy-buffer-size: "32k"

仍然獲得502。

不知道如何通過 kubectl 訪問 nginx.conf，從文檔看來，如果我更新此 ConfigMap，nginx 中的設置無論如何都會更改。

任何幫助表示贊賞。

更新

nginx.conf

# Configuration checksum: 1961171210939107273

# setup custom paths that do not require root access
pid /tmp/nginx.pid;

daemon off;

worker_processes 2;

worker_rlimit_nofile 523264;

worker_shutdown_timeout 240s ;

events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    client_max_body_size 100M;

    lua_package_path "/etc/nginx/lua/?.lua;;";
    
    lua_shared_dict balancer_ewma 10M;
    lua_shared_dict balancer_ewma_last_touched_at 10M;
    lua_shared_dict balancer_ewma_locks 1M;
    lua_shared_dict certificate_data 20M;
    lua_shared_dict certificate_servers 5M;
    lua_shared_dict configuration_data 20M;
    
    init_by_lua_block {
        collectgarbage("collect")
        
        -- init modules
        local ok, res
        
        ok, res = pcall(require, "lua_ingress")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        lua_ingress = res
        lua_ingress.set_config({
            use_forwarded_headers = false,
            use_proxy_protocol = false,
            is_ssl_passthrough_enabled = false,
            http_redirect_code = 308,
        listen_ports = { ssl_proxy = "442", https = "443" },
            
            hsts = true,
            hsts_max_age = 15724800,
            hsts_include_subdomains = true,
            hsts_preload = false,
        })
        end
        
        ok, res = pcall(require, "configuration")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        configuration = res
        end
        
        ok, res = pcall(require, "balancer")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        balancer = res
        end
        
        ok, res = pcall(require, "monitor")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        monitor = res
        end
        
        ok, res = pcall(require, "certificate")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        certificate = res
        end
        
        ok, res = pcall(require, "plugins")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        plugins = res
        end
        -- load all plugins that'll be used here
    plugins.init({})
    }
    
    init_worker_by_lua_block {
        lua_ingress.init_worker()
        balancer.init_worker()
        
        monitor.init_worker()
        
        plugins.run()
    }
    
    geoip_country       /etc/nginx/geoip/GeoIP.dat;
    geoip_city          /etc/nginx/geoip/GeoLiteCity.dat;
    geoip_org           /etc/nginx/geoip/GeoIPASNum.dat;
    geoip_proxy_recursive on;
    
    aio                 threads;
    aio_write           on;
    
    tcp_nopush          on;
    tcp_nodelay         on;
    
    log_subrequest      on;
    
    reset_timedout_connection on;
    
    keepalive_timeout  75s;
    keepalive_requests 100;
    
    client_body_temp_path           /tmp/client-body;
    fastcgi_temp_path               /tmp/fastcgi-temp;
    proxy_temp_path                 /tmp/proxy-temp;
    ajp_temp_path                   /tmp/ajp-temp;
    
    client_header_buffer_size       1M;
    client_header_timeout           60s;
    large_client_header_buffers     4 5M;
    client_body_buffer_size         1M;
    client_body_timeout             60s;
    
    http2_max_field_size            1M;
    http2_max_header_size           5M;
    http2_max_requests              1000;
    http2_max_concurrent_streams    128;
    
    types_hash_max_size             2048;
    server_names_hash_max_size      1024;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            64;
    
    proxy_headers_hash_max_size     512;
    proxy_headers_hash_bucket_size  64;
    
    variables_hash_bucket_size      256;
    variables_hash_max_size         2048;
    
    underscores_in_headers          off;
    ignore_invalid_headers          on;
    
    limit_req_status                503;
    limit_conn_status               503;
    
    include /etc/nginx/mime.types;
    default_type text/html;
    
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component;
    gzip_proxied any;
    gzip_vary on;
    
    # Custom headers for response
    
    server_tokens on;
    
    # disable warnings
    uninitialized_variable_warn off;
    
    # Additional available variables:
    # $namespace
    # $ingress_name
    # $service_name
    # $service_port
    log_format upstreaminfo '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';
    
    map $request_uri $loggable {
        
        default 1;
    }
    
    access_log /var/log/nginx/access.log upstreaminfo  if=$loggable;
    
    error_log  /var/log/nginx/error.log notice;
    
    resolver 10.245.0.10 valid=30s;
    
    # See https://www.nginx.com/blog/websocket-nginx
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        
        # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
        ''               '';
        
    }
    
    # Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.
    # If no such header is provided, it can provide a random value.
    map $http_x_request_id $req_id {
        default   $http_x_request_id;
        
        ""        $request_id;
        
    }
    
    # Create a variable that contains the literal $ character.
    # This works because the geo module will not resolve variables.
    geo $literal_dollar {
        default "$";
    }
    
    server_name_in_redirect off;
    port_in_redirect        off;
    
    ssl_protocols TLSv1.2;
    
    ssl_early_data off;
    
    # turn on session caching to drastically improve performance
    
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # allow configuring ssl session tickets
    ssl_session_tickets on;
    
    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;
    
    # allow configuring custom ssl ciphers
    ssl_ciphers '';
    ssl_prefer_server_ciphers on;
    
    ssl_ecdh_curve auto;
    
    # PEM sha: ---
    ssl_certificate     /etc/ingress-controller/ssl/default-fake-certificate.pem;
    ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem;
    
    proxy_ssl_session_reuse on;
    
    upstream upstream_balancer {
        ### Attention!!!
        #
        # We no longer create "upstream" section for every backend.
        # Backends are handled dynamically using Lua. If you would like to debug
        # and see what backends ingress-nginx has in its memory you can
        # install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.
        # Once you have the plugin you can use "kubectl ingress-nginx backends" command to
        # inspect current backends.
        #
        ###
        
        server 0.0.0.1; # placeholder
        
        balancer_by_lua_block {
            balancer.balance()
        }
        
        keepalive 32;
        
        keepalive_timeout  60s;
        keepalive_requests 100;
        
    }
    
    # Cache for internal auth checks
    proxy_cache_path /tmp/nginx-cache-auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=30m use_temp_path=off;
    
    # Global filters
    
    ## start server _
    server {
        server_name _ ;
        
        listen 80 default_server reuseport backlog=511 ;
        listen [::]:80 default_server reuseport backlog=511 ;
        listen 443 default_server reuseport backlog=511 ssl http2 ;
        listen [::]:443 default_server reuseport backlog=511 ssl http2 ;
        
        set $proxy_upstream_name "-";
        
        ssl_certificate_by_lua_block {
            certificate.call()
        }
        
        location / {
            
            set $namespace      "";
            set $ingress_name   "";
            set $service_name   "";
            set $service_port   "";
            set $location_path  "/";
            
            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = false,
                    ssl_redirect = false,
                    force_no_ssl_redirect = false,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }
            
            # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
            # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
            # other authentication method such as basic auth or external auth useless - all requests will be allowed.
            #access_by_lua_block {
            #}
            
            header_filter_by_lua_block {
                lua_ingress.header()
                plugins.run()
            }
            
            body_filter_by_lua_block {
            }
            
            log_by_lua_block {
                balancer.log()
                
                monitor.call()
                
                plugins.run()
            }
            
            access_log off;
            
            port_in_redirect off;
            
            set $balancer_ewma_score -1;
            set $proxy_upstream_name "upstream-default-backend";
            set $proxy_host          $proxy_upstream_name;
            set $pass_access_scheme  $scheme;
            
            set $pass_server_port    $server_port;
            
            set $best_http_host      $http_host;
            set $pass_port           $pass_server_port;
            
            set $proxy_alternative_upstream_name "";
            
            client_max_body_size                    1m;
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $remote_addr;
            
            proxy_set_header X-Forwarded-For        $remote_addr;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         off;
            proxy_buffer_size                       5M;
            proxy_buffers                           4 5M;
            
            proxy_max_temp_file_size                1024M;
            
            proxy_request_buffering                 on;
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_timeout             0;
            proxy_next_upstream_tries               3;
            
            proxy_pass http://upstream_balancer;
            
            proxy_redirect                          off;
            
        }
        
        # health checks in cloud providers require the use of port 80
        location /healthz {
            
            access_log off;
            return 200;
        }
        
        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            
            allow 127.0.0.1;
            
            allow ::1;
            
            deny all;
            
            access_log off;
            stub_status on;
        }
        
    }
    ## end server _
    
    ## start server dev-api
    server {
        server_name dev-api ;
        
        listen 80  ;
        listen [::]:80  ;
        listen 443  ssl http2 ;
        listen [::]:443  ssl http2 ;
        
        set $proxy_upstream_name "-";
        
        ssl_certificate_by_lua_block {
            certificate.call()
        }
        
        location / {
            
            set $namespace      "development";
            set $ingress_name   "app-ingress";
            set $service_name   "app-api-svc";
            set $service_port   "80";
            set $location_path  "/";
            
            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = false,
                    ssl_redirect = true,
                    force_no_ssl_redirect = false,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }
            
            # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
            # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
            # other authentication method such as basic auth or external auth useless - all requests will be allowed.
            #access_by_lua_block {
            #}
            
            header_filter_by_lua_block {
                lua_ingress.header()
                plugins.run()
            }
            
            body_filter_by_lua_block {
            }
            
            log_by_lua_block {
                balancer.log()
                
                monitor.call()
                
                plugins.run()
            }
            
            port_in_redirect off;
            
            set $balancer_ewma_score -1;
            set $proxy_upstream_name "development-app-api-svc-80";
            set $proxy_host          $proxy_upstream_name;
            set $pass_access_scheme  $scheme;
            
            set $pass_server_port    $server_port;
            
            set $best_http_host      $http_host;
            set $pass_port           $pass_server_port;
            
            set $proxy_alternative_upstream_name "";
            
            client_max_body_size                    1024M;
            
            client_body_buffer_size                 50M;
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $remote_addr;
            
            proxy_set_header X-Forwarded-For        $remote_addr;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   50s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         off;
            proxy_buffer_size                       5M;
            proxy_buffers                           8 5M;
            
            proxy_max_temp_file_size                1024M;
            
            proxy_request_buffering                 on;
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_timeout             0;
            proxy_next_upstream_tries               3;
            
            proxy_pass http://upstream_balancer;
            
            proxy_redirect                          off;
            
        }
        
    }
    ## end server dev-api
    
.......

更新 2

來自 kubectl logs -n nginx-ingress-controller-XXX 命令的日志

127.0.0.1 - - [16/Jul/2020:10:11:14 +0000] "POST [入口/服務端點] HTTP/2.0" 502 4 "https://[client-host-name]/[client- path]" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36 Edg/83.0.478.58" 9351 0.659 [service-name-80] [] 10.244.1.72:80 14 0.652 502 7b7bdf8a9319e88c80ba3444372daf2d

Answer 1

您需要確保在入口 controller 上設置了文件大小。 Nginx 將趕上設置。 嘗試這個。 有關注釋的更多信息，請遵循此。 https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: service-api-tls-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 8m

Answer 2

我最初的問題是 nginx，但在我更改限制后，它會將請求轉發到服務，但我沒有檢查正確的日志。 所以@mWatney 直接仔細檢查服務/吊艙是正確的。

對於其他任何人，我看到的問題與在 Linux alpine 容器中運行 .NET 核心 3.1 應用程序有關。 在應用程序中，我使用的是 System.Drawing.Common 版本，它會導致在 Linux 下運行異常，看起來像這樣；

System.TypeInitializationException：“Gdip”的類型初始化程序引發了異常。 ---> System.DllNotFoundException：無法加載共享庫“libgdiplus”

解決方案是添加到 dockerfile；

運行 apk 添加 libgdiplus-dev fontconfig ttf-dejavu --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --allow-untrusted

通過添加加載共享庫 libgdiplus 的功能，這允許在 Linux 下使用 System.Drawing.Common。

信用在這里： https://github.com/dotnet/dotnet-docker/issues/618#issuecomment-467619498

更持久的解決方案是完全從應用程序中刪除對 System.Drawing.Common 的依賴。 感謝@mWatney 的幫助，你讓我走上了正軌。