nginx tcp SYN packet not receiving ACK

Question

My server setup is nginx directly connects to a node.js server (nginx and node.js are in the same node and nginx is forwarding request to node: 127.0.0.1:8000). The symptom is sometimes there are some 504 logs in nginx log. And node.js log doesn't show any sign of ever receiving the request. I then enabled tcp log using iptables, which logs all tcp packets to port 8000. After checking the tcp log, it seems that nginx was trying to establish a tcp connection with node.js server, but it never succeeds. It just kept retrying sending SYN packets and then got timed out by nginx. Here's an example (tcp + nginx log):

13:44:44 sp:48103 dp:8000 SYN
13:44:45 sp:48103 dp:8000 SYN
13:44:47 sp:48103 dp:8000 SYN
13:44:51 sp:48103 dp:8000 SYN
13:44:59 sp:48103 dp:8000 SYN
13:45:15 sp:48103 dp:8000 SYN
13:45:44 nginx 504

During the period, the CPU load is pretty light, memory < 50%, incoming request is less than 50 per minute. And other requests were processed normally.

Server is Ubuntu 14.04.2 LTS

Any idea what's going on? Seems like an OS level issue? Thank you in advance.

Answer 1

Check whether something is actually running on TCP port 8000 on your loopback interface. Try some commands like:

lsof -P -n -i tcp:8000
fuser 8000/tcp
ss -4lnt
netstat -4lnt

These should give you some hints on whether something is listening at all. Or it may only be listening on a specific interface/address and not your loopback.