Play framework 2.7 with scala going down

Question

I have a play scala application running on play 2.7. this is used as a middleware for our frontend and it has rest end points.

Now I am running two different instances on cloud and using nginx and bound these two servers and load balance it with round robin.

Now I am having a problem that the servers goes down quite often ie 3 times a day and interesting thing is both server goes down at same time. When I looked at it says out of memory exception on the both servers. I tried to print javaheapdump for out of memory but getting no dump. I am still analysing the thread dump to figure out what might be the actual cause of my server going down but what pins me is why the two servers are going down at the same time.

Out of thread dump I see there are 7707 thread with sleeping state. here it is

"Connection evictor" #146 daemon prio=5 os_prio=0 cpu=2.33ms elapsed=1822.02s tid=0x00007f8a840c4800 nid=0x194 waiting on condition  [0x00007f8a58a5e000]
  java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(java.base@11/Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(java.base@11/Thread.java:834)

This what I see when server goes down

[35966.967s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Uncaught error from thread [application-akka.actor.default-dispatcher-1398Uncaught error from thread [application-akka.actor.default-dispatcher-1395]: ]: unable to create native thread: possibly out of memory or process/resource limits reachedunable to create native thread: possibly out of memory or process/resource limits reached, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[ ActorSystem[applicationapplication]
]
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
  at java.base/java.lang.Thread.start0(Native Method)
  at java.base/java.lang.Thread.start(Thread.java:803)
  at org.apache.http.impl.client.IdleConnectionEvictor.start(IdleConnectionEvictor.java:96)
  at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1219)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:298)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:236)
  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:223)
  at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:198)
  at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:934)
  at com.github.takezoe.solr.scala.SolrClient$.$anonfun$$lessinit$greater$default$2$1(SolrClient.scala:11)
  at com.github.takezoe.solr.scala.SolrClient.<init>(SolrClient.scala:14)
  at service.tvt.solr.SolrPolygonService.getSuburbBoundary(SolrPolygonService.scala:212)
  at service.tvt.search.OrbigoSearchService.mapfeeder(OrbigoSearchService.scala:54)
  at service.bto.business_categories.MeedssCountService.$anonfun$suburbMeedssCount$2(MeedssCountService.scala:81)
  at scala.collection.immutable.List.map(List.scala:287)
  at service.bto.business_categories.MeedssCountService.suburbMeedssCount(MeedssCountService.scala:80)
  at controllers.bto.industry_categories.meedss.MeedssController.$anonfun$suburbMeedssCount$1(MeedssController.scala:38)
  at play.api.mvc.ActionBuilder.$anonfun$apply$11(Action.scala:368)
  at scala.Function1.$anonfun$andThen$1(Function1.scala:52)
  at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:489)
  at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:487)
  at play.api.mvc.ActionBuilder$$anon$9.invokeBlock(Action.scala:336)
  at play.api.mvc.ActionBuilder$$anon$9.invokeBlock(Action.scala:331)
  at play.api.mvc.ActionBuilder$$anon$10.apply(Action.scala:426)
  at play.api.mvc.Action.$anonfun$apply$2(Action.scala:98)
  at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:184)
  at scala.util.Try$.apply(Try.scala:209)
  at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:184)
  at akka.stream.impl.Transform.apply(TraversalBuilder.scala:159)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:515)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:450)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:443)
  at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:629)
  at play.api.libs.streams.Accumulator$.$anonfun$futureToSink$2(Accumulator.scala:262)
  at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303)
  at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
  at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
  at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:72)
  at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
  at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallback(Promise.scala:312)
  at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:303)
  at scala.concurrent.impl.Promise.transformWith(Promise.scala:36)
  at scala.concurrent.impl.Promise.transformWith$(Promise.scala:34)
  at scala.concurrent.impl.Promise$DefaultPromise.transformWith(Promise.scala:183)
  at scala.concurrent.Future.flatMap(Future.scala:302)
  at scala.concurrent.Future.flatMap$(Future.scala:302)
  at scala.concurrent.impl.Promise$DefaultPromise.flatMap(Promise.scala:183)
  at play.api.libs.streams.Accumulator$.$anonfun$futureToSink$1(Accumulator.scala:261)
  at akka.stream.impl.Transform.apply(TraversalBuilder.scala:159)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:515)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:450)
  at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:443)
  at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:629)
  at play.api.libs.streams.SinkAccumulator.run(Accumulator.scala:144)
  at play.api.libs.streams.SinkAccumulator.run(Accumulator.scala:148)
  at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:441)
  at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
  at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51)
  at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
  at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
  at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
  at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
  at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
  at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
  at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
  at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Any quick pointers will be really helpful

Answer 1

Levi Ramsey was right it was because of TakeZoe lib which we were using. We were creating client for every new request and not closing it. Finally we created a connection pool with limited active connections and it worked.