Tomcat process time is small but nginx shows it is big

Question

I have five tomcat instances behind nginx.

Sometimes the nginx upstream_response_time is very big, more than 1 second, while the tomcat local access log shows the process time is only 50ms(I use %D to log process time).

What is the possible reason and how to fix it? It does not seems the network is slow since other applications run fast.

Update:

Seems the nginx upstream_response_time = %D + 1 sec .

Answer 1

My hypothesis for your observation are packet losses. This seems the most probable problem to me since you are saying that it happens when they are many requests. To test this, you can monitor the traffic eg with tcpdump or iftop . If you are under Ubuntu, you can install and start the latter with

sudo apt-get install iftop
sudo iftop

There are many other network monitoring solutions in Linux , the amazing Wireshark works for all operating systems.

One reason for package losses can be collisions, you can check that with a ifconfig [interface] if you are under Linux:

me@mymahine:~$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddrf f:41:8d:ef:41:8d  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 <-------------------------- check here ---
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Are the Tomcats and the Nginx on the same physical (or virtual) machine?

Further reading

• http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html

Answer 2

The servers normally queue the requests until a thread is available to handle it. If there are many requests in the queue but only a few threads, a single thread might handle the request quite fast, but if you add the time, the request was queued, the consumer sees a much longer time.

See: How to increase number of threads in tomcat thread pool?

Measuring the number of queued requests for tomcat

See if you can increase the number of threads or decrease the accept_count, but keep in mind, that the number of other resources like database connections might also need to be increased. Also keep in mind, that more threads might mean more competing for resources.

It might be worth to try change the parameters for this. Normally the access-log should also show the time the message is queue and handled, but I am not sure.