Dynamic fan out with Akka Streams

Question

I am building an Akka Streams application that goes through a couple of steps. There is one particular step that produces 0 or more results, it is not known in advance how many there are. Each of the results has to be processed asynchronously (by the same kind of component) and finally all the results have to be merged.

How should I model this in Akka Streams? I noticed that GraphDsl has a Broadcast element that lets you model a fan out, however this seems to be only possible when the number of outlets is know in advance. Is there a way in Akka Streams to have something like Broadcast but that fans out to a dynamic number of outlets?

Answer 1

Check out Hubs in this page: https://doc.akka.io/docs/akka/current/stream/stream-dynamic.html?language=scala

There are many cases when consumers or producers of a certain service (represented as a Sink, Source, or possibly Flow) are dynamic and not known in advance. The Graph DSL does not allow to represent this, all connections of the graph must be known in advance and must be connected upfront. To allow dynamic fan-in and fan-out streaming, the Hubs should be used.

Answer 2

It turns out that mapConcat does what I want. Here's a POC:

package streams

import scala.concurrent._
import akka._
import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
import scala.util.Random

object StreamsTest extends App {
  implicit val system = ActorSystem("TestSystem")
  implicit val materializer = ActorMaterializer()
  import system.dispatcher

  case class SplitRequest(s: String)

  def requestHandlerWithMultipleResults(request: SplitRequest): List[String] = 
   request.s.split(" ").toList

  def slowProcessingTask(s: String) =  {
    Thread.sleep(Random.nextInt(5000))
    s.toUpperCase
  }

  val g = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
    import GraphDSL.Implicits._

    val source: Source[String, NotUsed] = Source(List(SplitRequest("january february march april may")))
      .mapConcat(requestHandlerWithMultipleResults)
      .mapAsyncUnordered(5)(s => Future(slowProcessingTask(s)))

    val sink = Sink.foreach(println)

    source ~> sink

    ClosedShape
  })

  g.run()
}

Output, eg:

MAY
JANUARY
FEBRUARY
MARCH
APRIL