java web应用的高线程上下文切换

Question

We have been load testing our java web application and observe high cpu usage with 50 users (which doesn't seem practical). The CPU shoots up above 80%. While profiling it with java flight recording (JFR) we see that the context switch rate is 8400 per second (as seen in the Hot threads tab on java mission control). Analyzing the hot threads in jfr, it seems the cpu usage is distributed across the application threads with each thread using less than 3% cpu.

Increasing the user load to 100, 150 or 200 users we see the cpu shooting up above 90%, the throughput (transactions per second) remaining constant (as seen for 50 users load) while the response time crosses the acceptable threshold values (3 sec). Decreasing the user load to 20 users shows the cpu usage averages out to be above 55%. It certainly isn't true that the application threads are using up the cpu since our application is not a CPU bound application. The Hot Packages tab under Code tab group confirms this by showing that most of the time the application spends in is executing database queries.

We use glassfish 3.1.2.2 as our application server where the max thread pool is configured to be of 100. Oracle Linux Server release 6.4 is our operating system with linux kernel version as 2.6.39-400.214.4.el6uek.x86_64. I tried executing linux commands namely "watch -n0.5 pidstat -w -I -p " and "watch -n.5 grep ctxt /proc//status" to see the voluntary and involuntary thread context switching at OS level but they don't give any results.

Suspecting that high context switching could be causing the cpu to shoot up, do you have guidelines on what could be done to confirm that thread context switching is the cause of high cpu and what are there ways to tune the jvm or the application if that's the cause?

Thanks!

Answer 1

You can use performance counters to the number of context switch in a operation. In order to do so, uses the application perf.

The command should be perf stats -e cs <command> . This is an example:

[breno@debra ~]$ sudo perf stat  -e cs ls > /dev/null

Performance counter stats for 'ls':
   0    cs   (context switch)                                             

   0.001932855 seconds time elapsed

[breno@debra ~]$ sudo perf stat  -e cs ls -R > /dev/null

Performance counter stats for 'ls -R':

   3,130   cs (context switch)                                                        

   3.537120431 seconds time elapsed

Answer 2

I know it's an old one, but for the sake of whoever is dealing with the same issue.
The command: pidstat -wt 3, will give you the granularity of thread specific context switch. Then you can do a thread dump to your java process, And search for the thread number which you see high context switch. (you might need to translate the thread number to hex, depending on your thread dump output). We're still not sure what's the core problem though, cause the thread with the highest context switch is pointing to:

"NioBlockingSelector.BlockPoller-1" #37 daemon prio=5 os_prio=0 tid=0x00007f2b60b1f000 nid=0x1f48 runnable [0x00007f2b40af6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0000000700ae6c78> (a sun.nio.ch.Util$3)
        - locked <0x0000000700ae6c68> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0000000700ae6b30> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298)

And it makes sense, cause it's a thread selector, but not sure how to continue from here :)