Jetty maxes out number of Connectors

Question

We start the Jetty Server in Java and have a pretty straight forward process for that: API gets called, server starts. There will be no magic happening while starting Jetty, its a static process that shouldn't depend on anyhting. However we experience issues with some startups, where Jetty will try to open (presumably endlessly many) Connectors and maxes out the ThreadPool's size. OK-Runs will have exactly one Server Connector, as I would expect with a 4-core CPU and no Server Connector count set. We also tried to navigate around the problem by setting the number of Server Connectors (to one), still Jetty would ramp up the count and fail to start, because there would be not enough threads available. Even more curious, another API-User (different application) never had this issue once. This has all been tested on the same machine, same OS, often the same day even. We use Jetty 9.4.38. This is what the Exception says:

could not subscribe connector java.lang.IllegalStateException: Insufficient configured threads: required=200 < max=200 for QueuedThreadPool[qtp370296980]@16124894{STARTED,8<=144<=200,i=0,r=-1,q=0}[ReservedThreadExecutor@3825f21{s=0/6,p=0}] at org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:165) at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:141) at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:191) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

Answer 1

The ThreadPoolBudget being reported is indicating that you have some combination of technologies and/or components that are requiring a thread load of 200 threads, but your max threads is exactly 200.

The output from INFO logging of the named logger org.eclipse.jetty.util.thread.ThreadPoolBudget will report the lease / budget of the various components you have.

Output will have a format indicating ...

INFO: <component> requires <num> threads from <thread-pool>

That will tell you what components are requesting that amount of threads.

The <component> will include the hashcode / object identity of the component, if you see the same one repeated (eg: Foo@1234 and Foo@789 would be different object instances, but Bar@575 and Bar@575 would be the same object instance), then you have a bad initialization of your Server. (likely not using the LifeCycle properly)