Handle SIGTERM in akka-http

Question

The current (10.1.3) Akka HTTP docs:

https://doc.akka.io/docs/akka-http/current/server-side/graceful-termination.html

talk about graceful termination, using this code sample:

import akka.actor.ActorSystem
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.server.Route
import akka.stream.ActorMaterializer
import scala.concurrent.duration._

implicit val system = ActorSystem()
implicit val dispatcher = system.dispatcher
implicit val materializer = ActorMaterializer()

val routes = get {
  complete("Hello world!")
}

val binding: Future[Http.ServerBinding] =
    Http().bindAndHandle(routes, "127.0.0.1", 8080)

// ...
// once ready to terminate the server, invoke terminate:
val onceAllConnectionsTerminated: Future[Http.HttpTerminated] =
Await.result(binding, 10.seconds)
  .terminate(hardDeadline = 3.seconds)

// once all connections are terminated,
// - you can invoke coordinated shutdown to tear down the rest of the system:
onceAllConnectionsTerminated.flatMap { _ ⇒
  system.terminate()
}

I am wondering at what point this get called at, the comment states:

// once ready to terminate the server

What does this mean exactly, ie who/what determines the server is ready to terminate?

Do I have to put the shutdown code above in some hook function somewhere so that it is invoked on Akka HTTP receiving a SIGTERM?

I've tried putting this into the shutdown hook:

CoordinatedShutdown(system).addCancellableJvmShutdownHook{
  // once ready to terminate the server, invoke terminate:
  val onceAllConnectionsTerminated: Future[Http.HttpTerminated] =
  Await.result(binding, 10.seconds)
    .terminate(hardDeadline = 3.seconds)

  // once all connections are terminated,
  // - you can invoke coordinated shutdown to tear down the rest of the system:
  onceAllConnectionsTerminated.flatMap { _ ⇒
    system.terminate()
  }
}

But requests in progress are ended immediately upon sending a SIGTERM (kill ), rather than completing.

I also found a slightly different way of shutdown from https://github.com/akka/akka-http/issues/1210#issuecomment-338825745 :

CoordinatedShutdown(system).addTask(
    CoordinatedShutdown.PhaseServiceUnbind, "http_shutdown") { () =>

    bind.flatMap(_.unbind).flatMap { _ =>
      Http().shutdownAllConnectionPools
    }.map { _ =>
      Done
    }
  }

Maybe I should using this to handle SIGTERM? I'm not sure..

Thanks!

Answer 1

Resolution taken from this answer here:

https://discuss.lightbend.com/t/graceful-termination-on-sigterm-using-akka-http/1619

CoordinatedShutdown(system).addTask(
    CoordinatedShutdown.PhaseServiceUnbind, "http_shutdown") { () =>

    bind.flatMap(_.terminate(hardDeadline = 1.minute)).map { _ =>
      Done
    }
  }

For me the main part was to increase akka.coordinated-shutdown.default-phase-timeout as it took longer to finish the processing of the request than the 5 second default. You can also just increase the timeout for that one phase. I had the following message in my logs:

Coordinated shutdown phase [service-unbind] timed out after 5000 milliseconds