在Akka Streams中使用動態接收器目標

Question

我對Akka很新，我正在努力學習基礎知識。 我的用例是不斷從JMS隊列中讀取消息並將每條消息輸出到一個新文件。 我有基本的設置：

Source<String, NotUsed> jmsSource =
  JmsSource
    .textSource(JmsSourceSettings
    .create(connectionFactory)
    .withQueue("myQueue")
    .withBufferSize(10));

Sink<ByteString, CompletionStage<IOResult>> fileSink =
  FileIO.toFile(new File("random.txt"));

final Flow<String, ByteString, NotUsed> flow = Flow.fromFunction((String n) -> ByteString.fromString(n));

final RunnableGraph<NotUsed> runnable = jmsSource.via(flow).to(fileSink);

runnable.run(materializer);

但是，我希望文件名是動態的（而不是硬編碼為“random.txt”）：它應該根據隊列中每條消息的內容進行更改。 當然，我可以在流程中選擇文件名，但是如何在fileSink設置該名稱？ 我該如何最好地設置它？

Answer 1

我基於akka.stream.impl.LazySink創建了一個簡單的Sink。 我在成功案例中只用一個元素測試過它，所以請隨時在這里評論或者GitHub Gist 。

import akka.NotUsed
import akka.stream.{Attributes, Inlet, SinkShape}
import akka.stream.scaladsl.{Sink, Source}
import akka.stream.stage._

class OneToOneOnDemandSink[T, +M](sink: T => Sink[T, M]) extends GraphStage[SinkShape[T]] {

  val in: Inlet[T] = Inlet("OneToOneOnDemandSink.in")
  override val shape = SinkShape(in)

  override def createLogic(inheritedAttributes: Attributes) = new GraphStageLogic(shape) {

    override def preStart(): Unit = pull(in)

    val awaitingElementHandler = new InHandler {
      override def onPush(): Unit = {
        val element = grab(in)
        val innerSource = createInnerSource(element)
        val innerSink = sink(element)
        Source.fromGraph(innerSource.source).runWith(innerSink)(subFusingMaterializer)
      }

      override def onUpstreamFinish(): Unit = completeStage()

      override def onUpstreamFailure(ex: Throwable): Unit = failStage(ex)
    }
    setHandler(in, awaitingElementHandler)

    def createInnerSource(element: T): SubSourceOutlet[T] = {
      val innerSource = new SubSourceOutlet[T]("OneToOneOnDemandSink.innerSource")

      innerSource.setHandler(new OutHandler {
        override def onPull(): Unit = {
          innerSource.push(element)
          innerSource.complete()
          if (isClosed(in)) {
            completeStage()
          } else {
            pull(in)
            setHandler(in, awaitingElementHandler)
          }
        }

        override def onDownstreamFinish(): Unit = {
          innerSource.complete()
          if (isClosed(in)) {
            completeStage()
          }
        }
      })

      setHandler(in, new InHandler {
        override def onPush(): Unit = {
          val illegalStateException = new IllegalStateException("Got a push that we weren't expecting")
          innerSource.fail(illegalStateException)
          failStage(illegalStateException)
        }

        override def onUpstreamFinish(): Unit = {
          // We don't stop until the inner stream stops.
          setKeepGoing(true)
        }

        override def onUpstreamFailure(ex: Throwable): Unit = {
          innerSource.fail(ex)
          failStage(ex)
        }
      })

      innerSource
    }
  }
}

object OneToOneOnDemandSink {

  def apply[T, M](sink: T => Sink[T, M]): Sink[T, NotUsed] = Sink.fromGraph(new OneToOneOnDemandSink(sink))
}

這將為每個元素創建一個新的Sink，因此它避免了LazySink具有的大量復雜性，並且也沒有合理的物化價值。

Answer 2

以下是三種等效方法：

• 再簡潔一點，使用map和內部圖表，
• 再簡潔一點，使用flatMapConcat和內部圖形，
• 使用GraphDSL再詳細GraphDSL 。

在所有情況下，輸出是：

$ tail -n +1 -- *.txt
==> 1.txt <==
1

==> 2.txt <==
2

==> 3.txt <==
3

==> 4.txt <==
4

==> 5.txt <==
5

---

1. 使用map ：

    import java.nio.file.Paths import akka.actor.ActorSystem import akka.stream._ import akka.stream.scaladsl.{FileIO, Sink, Source} import akka.util.ByteString import scala.concurrent.Future object Example extends App { override def main(args: Array[String]): Unit = { implicit val system = ActorSystem("Example") implicit val materializer = ActorMaterializer() val result: Future[Seq[Future[IOResult]]] = Source(1 to 5) .map( elem => Source.single(ByteString(s"$elem\\n")) .runWith(FileIO.toPath(Paths.get(s"$elem.txt"))) ) .runWith(Sink.seq) implicit val ec = system.dispatcher result.onComplete(_ => system.terminate()) } } 

   說明 ：我們將Int元素map到一個函數，該函數創建一個內部圖形， Source.single(ByteString(...)).runWith(FileIO.toPath(...) ，它序列化並寫入動態路徑，並允許我們通過Sink.seq積累最終的Future[IOResult] 。

   文件 ：

   map通過調用映射函數並將返回值傳遞給下游來轉換流中的每個元素。

   映射函數返回元素時發出

   背壓時下游的背壓

   上游完成時完成

   另見 ：

   • https://doc.akka.io/docs/akka/current/stream/stages-overview.html#map

---

2. 使用flatMapConcat ：

    import java.nio.file.Paths import akka.actor.ActorSystem import akka.stream._ import akka.stream.scaladsl.{FileIO, Sink, Source} import akka.util.ByteString import scala.concurrent.Future object Example extends App { override def main(args: Array[String]): Unit = { implicit val system = ActorSystem("Example") implicit val materializer = ActorMaterializer() val result: Future[Seq[Future[IOResult]]] = Source(1 to 5) .flatMapConcat( elem => Source.single( Source.single(ByteString(s"$elem\\n")) .runWith(FileIO.toPath(Paths.get(s"$elem.txt"))) ) ) .runWith(Sink.seq) implicit val ec = system.dispatcher result.onComplete(_ => system.terminate()) } } 

   說明 ： flatMapConcat需要Source 。 因此，我們創造一個發射所述mat內的圖形， Source.single(ByteString(...)).runWith(FileIO.toPath(...)這使我們能夠累積所得的Future[IOResult]通過Sink.seq 。實際的序列化和調度由內部圖形完成。

   文件 ：

   flatMapConcat將每個輸入元素轉換為Source ，然后通過連接將其元素展平為輸出流。 這意味着在消耗下一個源開始之前，每個源都被完全消耗。

   當前消耗的子流具有可用元素時發出

   背壓時下游的背壓

   完成時上游和完成全部耗盡子完整

   另見 ：

   • https://doc.akka.io/docs/akka/current/stream/stages-overview.html#flatmapconcat
   • https://doc.akka.io/docs/akka/current/stream/stream-substream.html#flatmapconcat

---

3. 使用GraphDSL自定義Sink器：

    import java.nio.file.Path import akka.stream.scaladsl.{Broadcast, FileIO, Flow, GraphDSL, Sink, Source, ZipWith} import akka.stream.{IOResult, Materializer, SinkShape} import akka.util.ByteString import scala.concurrent.Future object FileSinks { def dispatch[T]( dispatcher: T => Path, serializer: T => ByteString )( implicit materializer: Materializer ): Sink[T, Future[Seq[Future[IOResult]]]] = Sink.fromGraph( GraphDSL.create( Sink.seq[Future[IOResult]] ) { implicit builder => sink => // prepare this sink's graph elements: val broadcast = builder.add(Broadcast[T](2)) val serialize = builder.add(Flow[T].map(serializer)) val dispatch = builder.add(Flow[T].map(dispatcher)) val zipAndWrite = builder.add(ZipWith[ByteString, Path, Future[IOResult]]( (bytes, path) => Source.single(bytes).runWith(FileIO.toPath(path))) ) // connect the graph: import GraphDSL.Implicits._ broadcast.out(0) ~> serialize ~> zipAndWrite.in0 broadcast.out(1) ~> dispatch ~> zipAndWrite.in1 zipAndWrite.out ~> sink // expose ports: SinkShape(broadcast.in) } ) } ---- import java.nio.file.Paths import FileSinks import akka.actor.ActorSystem import akka.stream._ import akka.stream.scaladsl.Source import akka.util.ByteString import scala.concurrent.Future object Example extends App { override def main(args: Array[String]): Unit = { implicit val system = ActorSystem("Example") implicit val materializer = ActorMaterializer() val result: Future[Seq[Future[IOResult]]] = Source(1 to 5) .runWith(FileSinks.dispatch[Int]( elem => Paths.get(s"$elem.txt"), elem => ByteString(s"$elem\\n")) ) implicit val ec = system.dispatcher result.onComplete(_ => system.terminate()) } } 

   • dispatcher是一個將輸入對象轉換為路徑的函數，您可以在此處動態決定路徑。
   • serializer是一個序列化輸入對象的函數。
   • 其余的使用GraphDSL ，另見： https ： GraphDSL

---

免責聲明 ：我本人仍在學習Akka Stream。