Tomcat线程转储-可能卡住了线程

Question

I ran three thread dumps, 10 seconds apart, on a production server running Tomcat. I found that the same thread (same thread id) appears in all three thread dumps and is doing exactly the same thing - processing a JSP page. Does this mean it is stuck? Or could it be that the same thread from the Tomcat thread pool keeps getting assigned the same task over and over again.

This is a moderately big application and has a lot of servlets/JSP pages, business logic and a database. Also, it is fairly busy (around 20 requests per second). Tomcat is running standalone (not behind Apache).

Edit 1 (additional info): The home page normally takes significantly less than 10 seconds to render, so this can't be a typical single request to the home page.

Edit 2: I took another thread dump hours later (but without restarting Tomcat) and this specific thread was doing something else. Does this CONCLUSIVELY mean that the thread was not stuck?

Here is the the section of the thread dump for that specific thread (only one is listed as the other two are exactly the same):

"http-bio-8080-exec-220" daemon prio=10 tid=0x00007f5c801de800 nid=0x3151 runnable [0x00007f5c74d28000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
    at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
    at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
    at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119)
    at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
    at org.apache.coyote.Response.doWrite(Response.java:517)
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
    at org.apache.catalina.connector.OutputBuffer.realWriteChars(OutputBuffer.java:487)
    at org.apache.tomcat.util.buf.CharChunk.append(CharChunk.java:312)
    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:528)
    at org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:152)
    at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
    at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:336)
    at java.io.Writer.write(Writer.java:157)
    at org.apache.jasper.runtime.JspWriterImpl.print(JspWriterImpl.java:473)
    at org.apache.taglibs.standard.tag.common.core.UrlSupport.doEndTag(UrlSupport.java:127)
    at org.apache.jsp.WEB_002dINF.jsp.site.home_jsp._jspx_meth_c_005furl_005f22(home_jsp.java:1172)
    at org.apache.jsp.WEB_002dINF.jsp.site.home_jsp._jspx_meth_c_005fif_005f2(home_jsp.java:1141)
    at org.apache.jsp.WEB_002dINF.jsp.site.home_jsp._jspx_meth_c_005fforEach_005f1(home_jsp.java:1102)
    at org.apache.jsp.WEB_002dINF.jsp.site.home_jsp._jspService(home_jsp.java:308)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
    at inspirasites.servlets.HomeServlet.doGet(HomeServlet.java:102)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at inspirasites.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:23)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at inspirasites.filters.SiteFilter.doFilter(SiteFilter.java:45)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    - locked <0x00000000e12fb330> (a org.apache.tomcat.util.net.SocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

Answer 1

I have recently seen the exact same behaviour in our application and was able to reproduce it. It can be caused by a mis-behaving client that sends a request but does not consume the response data. In this case the socket will block, see the answers for this question: Is an OutputStream in Java blocking? (Sockets)