Spring Boot REST API server Connection timed out sometime

Question

I'm testing my Spring Boot REST API server with Apache JMeter.

Most of 'Connect Time' is in 10ms, but connection timed out occurs sometime.

JMeter's settings

- Thread
Number of Threads : 500
Ramp-Up Period : 50

- HTTP Request
Connect timeout : 20000ms
Response timeout : 40000ms

- Constant timer
Thread delay : 1000ms

CentOS7 settings

open files                      (-n) 65535
max user processes              (-u) 65535

Java execution settings

-Xmx8g -Xms4g -Xmn2g -XX:PermSize=512m -XX:SurvivorRatio=16

Spring Boot application.properties Tomcat settings

server.tomcat.connection-timeout=20000
server.tomcat.max-connections=100000
server.tomcat.threads.max=10000
server.tomcat.threads.min-spare=2500
server.tomcat.accept-count=10000

JMeter works normally up to 10,000~20,000 requests, but connection timeout occurs after that.

Starting standalone test @ Fri Apr 16 09:30:29 KST 2021 (1618533029412)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary +      1 in 00:00:03 =    0.4/s Avg:  1319 Min:  1319 Max:  1319 Err:     0 (0.00%) Active: 127 Started: 127 Finished: 0
summary +   8771 in 00:00:28 =  318.5/s Avg:   398 Min:   236 Max:  4191 Err:     0 (0.00%) Active: 500 Started: 500 Finished: 0
summary =   8772 in 00:00:30 =  290.3/s Avg:   398 Min:   236 Max:  4191 Err:     0 (0.00%)
summary +   9349 in 00:00:30 =  311.6/s Avg:   534 Min:   237 Max:  3582 Err:     0 (0.00%) Active: 500 Started: 500 Finished: 0
summary =  18121 in 00:01:00 =  300.9/s Avg:   468 Min:   236 Max:  4191 Err:     0 (0.00%)
summary +   5872 in 00:00:30 =  195.7/s Avg:  1494 Min:   235 Max: 20855 Err:    10 (0.17%) Active: 500 Started: 500 Finished: 0
summary =  23993 in 00:01:30 =  265.9/s Avg:   719 Min:   235 Max: 20855 Err:    10 (0.04%)

org.apache.http.conn.HttpHostConnectException: Connect to <IP Address>:<Port> [/112.220.184.107] failed: Connection timed out: connect
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$JMeterDefaultHttpClientConnectionOperator.connect(HTTPHC4Impl.java:336)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:843)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:574)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:67)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1231)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1220)
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:622)
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:546)
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:486)
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:253)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection timed out: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    ... 19 more

Can you tell me why it happens?

Answer 1

summary +   5872 in 00:00:30 =  195.7/s Avg:  1494 Min:   235 Max: 20855 Err:    10 (0.17%) Active: 500 Started: 500 Finished: 0
                                                                  ^^^^^^^

Your response time increases above 20 seconds hence Tomcat rejects the connection (most probably the setting acts as the protection from slow HTTP attacks

either set it to -1 to disable it or use a profiler tool to see what is causing the problem on your application/tomcat end.

Also JMeter command-line output doesn't tell the full story, I would recommend generating some charts from the.jtl results file like Response Times Over Time , Active Threads Over Time , Transactions per Second , etc. so you could correlate various performance metrics. It's also a good idea to monitor server-side operating system health metrics like CPU, RAM, Network, Swap, Disk usage as well as JVM JMX metrics, it can be done using ie JMeter PerfMon Plugin