Spring boot thread hanging?

Question

I have a simple (thats what I think) Spring boot application. There are 4 layers:

1. Rest Controller
2. Application Service (called by the Rest Controller)
3. Domain Service (called by Application Service. It connects to the database - repository layer)
4. Adapter Service (called by Application Service for outbound calls via Hystrix)

Now the problem is that it can only handle a max of 15 parallel calls. If any additional REST API request arrives while these calls are being processed, it makes it to the Application Service layer and then waits. Once one of those 15 parallel call returns, then the new request proceeds to make call to the Domain Service Layer and return.

I tried multiple things:

1. Increasing spare threads for the server in application.properties file

    server.tomcat.min-spare-threads=1000 server.tomcat.max-connections=1000 server.tomcat.max-threads=1000 

Once I do this, I see the # of http-nio-* threads increase to 1000 but the hanging issue is not fixed.

2. I found this snippet online to customize the tomcat container but it didn't help either:

    @Bean public WebServerFactoryCustomizer<TomcatServletWebServerFactory> containerCustomizer() { return new WebServerFactoryCustomizer<TomcatServletWebServerFactory>() { @Override public void customize(TomcatServletWebServerFactory factory) { factory.addConnectorCustomizers(new TomcatConnectorCustomizer() { @Override public void customize(Connector connector) { Arrays.stream(connector.getProtocolHandler().findUpgradeProtocols()) .filter(upgradeProtocol -> upgradeProtocol instanceof Http2Protocol) .map(upgradeProtocol -> (Http2Protocol) upgradeProtocol) .forEach(http2Protocol -> { http2Protocol.setMaxConcurrentStreamExecution(1000); }); } }); } }; } 

3. I tried configuring the thread pool via code

    @Bean(name = "taskExecutor") public TaskExecutor threadPoolTaskExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor.setCorePoolSize(200); executor.setMaxPoolSize(300); executor.setQueueCapacity(300); executor.setThreadNamePrefix("anniversary"); executor.initialize(); System.out.println("******* name " + executor.getThreadNamePrefix()); System.out.println("********** core pool size " + executor.getCorePoolSize()); return executor; } 

But none of this helps and I believe the issue is not with the number of threads but elsewhere since the request is not able to go from one service to another. There are hundreds of http-nio-* threads in waiting state and when a new request comes in, its assigned its own thread and I can see that in the Debug mode.

Any pointers, help, tips are much appreciated. What resource is required for service to service invocation by Spring boot?

Answer 1

I believe your observation is right - it's most likely not tomcat who's the bottleneck here. From what you write, would rather look at the domain service. Is the domain service doing some communication with the database or talking to something else over the network (for example over HTTP)?

If you happen to do database in there, check for spring's datasource configuration. There is going to be a database connection pool with a limited number of maximum concurrent connections to the database. Once these connections are all in use, threads that want to talk to the DB will be blocked until one of the connection becomes free again.

Similar connection pools are in place with many other things that talk over network (eg Apache HTTP Client also has a connection pool that can be configured).

That's where i would look next.

Cheers, Matthias