來自 AWS 應用程序負載均衡器的隨機 502 錯誤

Question

我們收到來自 ALB 的隨機 502 錯誤，我們的后端根本沒有受到攻擊，因為沒有請求日志。 ALB 中沒有任何內容僅記錄 502 錯誤，但沒有任何可用於調試的內容。

h2 2020-03-26T14:30:52.495547Z app/path/tomytarget 10.111.11.111:50103 100.00.00.00:8080:8080 0.001 18.799 -1 502 - 1213 208 "POST https://mydomain:443/user/auth HTTP/2.0" "Name/3 CFNetwork/1121.2.2 Darwin/19.2.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:ap-southeast-1:0000000000:targetgroup/path/tomytarget "Root=someId" "mydomain.com" "arn:aws:acm:ap-southeast-1:0000000000:certificate/certificatedId" 0 2020-03-26T14:30:33.694000Z "forward" "-" "-" "100.00.00.00:8080" "-"

在 nodejs 和 express 中使用正確的路由啟用健康檢查后，我們開始注意到它

app.get("/health-check", (req, res) => {
  res.status(200).end();
});

這是我們的 ALB 配置，我們使用 VPC 對等連接到另一個 VPC

  ElasticLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      IpAddressType: ipv4
      Scheme: internet-facing
      SecurityGroups:
        - !Ref ELBSecurityGroup
      Subnets:
        - !Ref PublicSubnetA
        - !Ref PublicSubnetB
      Type: application

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 8080
      Protocol: HTTP
      Targets:
        - Id: <some ip in the other VPC>
          AvailabilityZone: all
          Port: 8080
      TargetType: ip
      VpcId: !Ref VPC
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: /health-check
      HealthCheckPort: 8080
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 3
      UnhealthyThresholdCount: 5

  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref TargetGroup
      LoadBalancerArn: !Ref ElasticLoadBalancer
      Certificates:
        - CertificateArn: !Ref CertificateArn
      Port: 443
      Protocol: HTTPS

正如我所說，我們正在使用 VPC 對等和 HTTPS 以及來自 AWS 證書管理器的證書

編輯：

如果您使用的是 nodejs，一種解決方案可能如下

// AWS ALB keepAlive is set to 60 seconds, we need to increase the default KeepAlive timeout
// of our node server
server.keepAliveTimeout = 65000; // Ensure all inactive connections are terminated by the ALB, by setting this a few seconds higher than the ALB idle timeout
server.headersTimeout = 66000; // Ensure the headersTimeout is set higher than the keepAliveTimeout due to this nodejs regression bug: https://github.com/nodejs/node/issues/27363

Answer 1

我已經在nodejs中解決了這個問題

// AWS ALB keepAlive is set to 60 seconds, we need to increase the default KeepAlive timeout
// of our node server
server.keepAliveTimeout = 65000; // Ensure all inactive connections are terminated by the ALB, by setting this a few seconds higher than the ALB idle timeout
server.headersTimeout = 66000; // Ensure the headersTimeout is set higher than the keepAliveTimeout due to this nodejs regression bug: https://github.com/nodejs/node/issues/27363