Tomcat 9 slow response

Question

I have a web application(jsp) which was running fine on Tomcat 8.0.46 for more than a year. Few weeks back we upgraded to Tomcat 9.0.10, after couple of days of upgrade tomcat is responding with a delay of 8-16 seconds for some of the request.

I saw more than 800 request/sec in localhostaccess log, so I increased maxThreads to 512 as below and max heap memory to 4096MB.

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="512" minSpareThreads="4"/>

<Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

But the issue was not resolved, so I compared all the configuration with old Tomcat and found that tomcat9 is using tomcat executor where old was not using it. will executor impact request handling time?

Old tomcat configuration

<Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

FYI the webapp consist of only jsps and few of them used to interact with DB using DBCP and gives XML response.

I am not suspecting DB connection pool because it was already used and no change.

CPU : Xenon
RAM: 8GB
OS : Window 2012 server
JDK : jdk1.8.0_144

Added response time duration to localhost access log, can see the delay in some request, but the request prior and after are having quick response withing 15 milliseconds(bold).

    10.50.29.27 - - [17/Dec/2018:09:27:23 -0500] "GET /App1/sendevent.jsp?TNAME=Transfer1 HTTP/1.1" 200 90 270BA450469B7AA71D22252711CA288A **0.015** http-nio-8080-exec-3
    10.50.29.26 - - [17/Dec/2018:09:27:23 -0500] "GET /App1/Start.jsp?ACTION=START&ID=3154583920&SID=$num$&SESSIONID=63AA673E-B6EF-447E-AAB9-3B5B7260EB03&ScriptID=$sid$&ScriptData=$scriptdata$ HTTP/1.1" 200 2948 D97741884AD1005359430A3307D5D44E **6.031** http-nio-8080-exec-5
    10.50.29.27 - - [17/Dec/2018:09:27:23 -0500] "GET /App1/sendevent.jsp?TNAME=Transfer1&TRANSFER_RESULT=S&LAST_ACTION=1&TRANSFER_REASON=connection.disconnect.transfer&TRANSFER_NOTE=undefined HTTP/1.1" 200 90 270BA450469B7AA71D22252711CA288A **0.000** http-nio-8080-exec-9

Answer 1

acceptorThreadCount=2 solved productivity problems for me in two cases:

1. Tomcat 8 under Debian on a small virtual machine (application is XWiki)
2. Tomcat 9 under Centos on a big production server (application is DSpace:jspui/xmlui/oai/solr).

The third case when I saw the sufficiently better productivity is:

3. Tomcat 8 under Windows Server 2008 Standard SP2 on a very small old Dell server (application is DSpace:jspui/xmlui/oai/solr). It is the previous instance of Case 2 kept until end of transition.