Nginx returns Bad Gateway 502 in Kubernetes

Question

Environment: RubyOnRails + Puma + nginx + Docker + Kubernetes

I made a deployment in k8s. I made one pod including 3 containers.

One for ruby on rails, another for webpacker, and the other for nginx.

But the problem is that when I check the localhost in my nginx container by curl http://localhost it returns:

502 bad gateway

There is no problem with the result from curl http://localhost:3000 .

I think there might be something wrong in nginx settings.

Please let me know what the problem is.

Kubernetes yaml

apiVersion: v1
kind: Service
metadata:
  namespace: line-manager
  name: web
  labels:
    app: web
spec:
  selector:
    app: web
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: line-manager
  name: web
spec:
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: khjoo19/nginx:v2
        imagePullPolicy: Always
        command: ["/usr/sbin/nginx", "-g", "daemon off;", "-c", "/etc/nginx/nginx.conf"]
        ports:
        - containerPort: 80
        volumeMounts:
        - name: mysql-pv
          mountPath: /var/lib/mysql

      - name: webpacker
        image: khjoo19/fullout-line:v2
        imagePullPolicy: Always
        command: ["bundle", "exec", "bin/webpack-dev-server"]
        ports:
        - containerPort: 8080

      - name: web
        image: khjoo19/fullout-line:v2
        imagePullPolicy: Always
        command: ["bundle", "exec", "puma", "-C", "config/puma.rb"]
        env:
        - name: MYSQL_DATABASE
          value: lineManage_db
        - name: MYSQL_USER
          value: root
        - name: MYSQL_HOST
          value: mysql
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3000
        readinessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 30
          timeoutSeconds: 30
        volumeMounts:
        - name: mysql-pv
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-pv
        persistentVolumeClaim:
          claimName: mysql-pvc

nginx.conf

upstream line_manager {
  server unix:///line_manager/tmp/sockets/puma.sock;
}

server {
  listen 80;
  server_name localhost;

  access_log /var/log/nginx/access.log;
  error_log  /var/log/nginx/error.log;

  root /line_manager/public;

  client_max_body_size 100m;
  error_page 404             /404.html;
  error_page 505 502 503 504 /500.html;
  try_files  $uri/index.html $uri @line_manager;
  keepalive_timeout 30;

  location @line_manager {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_pass http://line_manager;
  }
}

puma.rb

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }.to_i
threads threads_count, threads_count
port        ENV.fetch("PORT") { 3000 }
environment ENV.fetch("RAILS_ENV") { "development" }
plugin :tmp_restart

app_root = File.expand_path("../..", __FILE__)
bind "unix://#{app_root}/tmp/sockets/puma.sock"

stdout_redirect "#{app_root}/log/puma.stdout.log", "#{app_root}/log/puma.stderr.log", true

Answer 1

I ran a pod using your nginx image, and this is what I get:

~# kubectl run demo --image khjoo19/nginx:v2 -- sh -c "/usr/sbin/nginx -g 'daemon off;' -c /etc/nginx/nginx.conf"
deployment.apps/demo created
root@v1-16-master:~# kubectl get po -owide
NAME                      READY   STATUS    RESTARTS   AGE     IP                NODE             NOMINATED NODE   READINESS GATES
curler-755cc7cfff-qjl2n   1/1     Running   5          6d1h    192.168.113.161   v1-16-worker-1   <none>           <none>
debian-68649788ff-v6lrc   1/1     Running   5          5d21h   192.168.49.31     v1-16-worker-2   <none>           <none>
demo-86f65d889-rfbzx      1/1     Running   0          21s     192.168.49.62     v1-16-worker-2   <none>           <none>
root@v1-16-master:~# kubectl exec curler-755cc7cfff-qjl2n -- curl 192.168.49.62 2>/dev/null
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>
root@v1-16-master:~# kubectl logs demo-86f65d889-rfbzx
2020/01/23 10:27:30 [crit] 9#9: *1 connect() to unix:///line_manager/tmp/sockets/puma.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.113.161, server: localhost, request: "GET / HTTP/1.1", upstream: "http://unix:///line_manager/tmp/sockets/puma.sock:/", host: "192.168.49.62"
2020/01/23 10:27:30 [crit] 9#9: *1 connect() to unix:///line_manager/tmp/sockets/puma.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.113.161, server: localhost, request: "GET / HTTP/1.1", upstream: "http://unix:///line_manager/tmp/sockets/puma.sock:/500.html", host: "192.168.49.62"
192.168.113.161 - - [23/Jan/2020:10:27:30 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.59.0"

Your nginx image is missing the file descriptor. And I don't see anywhere in your Deployment you are covering this.

You would need to create an emptyDir volume and map /line_manager/tmp/sockets/ directory with the directory where is you puma.sock socket descriptor; on the other container. Should work.