[英]Using dynamic sink destination in Akka Streams
I'm rather new to Akka and am trying to learn the basics. 我对Akka很新,我正在努力学习基础知识。 My use case is to continually read messages from a JMS queue and output each message to a new file.
我的用例是不断从JMS队列中读取消息并将每条消息输出到一个新文件。 I have the basic setup working with:
我有基本的设置:
Source<String, NotUsed> jmsSource =
JmsSource
.textSource(JmsSourceSettings
.create(connectionFactory)
.withQueue("myQueue")
.withBufferSize(10));
Sink<ByteString, CompletionStage<IOResult>> fileSink =
FileIO.toFile(new File("random.txt"));
final Flow<String, ByteString, NotUsed> flow = Flow.fromFunction((String n) -> ByteString.fromString(n));
final RunnableGraph<NotUsed> runnable = jmsSource.via(flow).to(fileSink);
runnable.run(materializer);
However, I want the file name to be dynamic (and not hard-coded to "random.txt"): it should be changed depending on the content of each message on the queue. 但是,我希望文件名是动态的(而不是硬编码为“random.txt”):它应该根据队列中每条消息的内容进行更改。 I could, of course, pick up the file name in the flow, but how do I set that name in
fileSink
? 当然,我可以在流程中选择文件名,但是如何在
fileSink
设置该名称? How do I best set this up? 我该如何最好地设置它?
I created a simple Sink based on akka.stream.impl.LazySink
. 我基于
akka.stream.impl.LazySink
创建了一个简单的Sink。 I have only tested it with a single element in the successful case so feel free to comment on here or the GitHub Gist . 我在成功案例中只用一个元素测试过它,所以请随时在这里评论或者GitHub Gist 。
import akka.NotUsed
import akka.stream.{Attributes, Inlet, SinkShape}
import akka.stream.scaladsl.{Sink, Source}
import akka.stream.stage._
class OneToOneOnDemandSink[T, +M](sink: T => Sink[T, M]) extends GraphStage[SinkShape[T]] {
val in: Inlet[T] = Inlet("OneToOneOnDemandSink.in")
override val shape = SinkShape(in)
override def createLogic(inheritedAttributes: Attributes) = new GraphStageLogic(shape) {
override def preStart(): Unit = pull(in)
val awaitingElementHandler = new InHandler {
override def onPush(): Unit = {
val element = grab(in)
val innerSource = createInnerSource(element)
val innerSink = sink(element)
Source.fromGraph(innerSource.source).runWith(innerSink)(subFusingMaterializer)
}
override def onUpstreamFinish(): Unit = completeStage()
override def onUpstreamFailure(ex: Throwable): Unit = failStage(ex)
}
setHandler(in, awaitingElementHandler)
def createInnerSource(element: T): SubSourceOutlet[T] = {
val innerSource = new SubSourceOutlet[T]("OneToOneOnDemandSink.innerSource")
innerSource.setHandler(new OutHandler {
override def onPull(): Unit = {
innerSource.push(element)
innerSource.complete()
if (isClosed(in)) {
completeStage()
} else {
pull(in)
setHandler(in, awaitingElementHandler)
}
}
override def onDownstreamFinish(): Unit = {
innerSource.complete()
if (isClosed(in)) {
completeStage()
}
}
})
setHandler(in, new InHandler {
override def onPush(): Unit = {
val illegalStateException = new IllegalStateException("Got a push that we weren't expecting")
innerSource.fail(illegalStateException)
failStage(illegalStateException)
}
override def onUpstreamFinish(): Unit = {
// We don't stop until the inner stream stops.
setKeepGoing(true)
}
override def onUpstreamFailure(ex: Throwable): Unit = {
innerSource.fail(ex)
failStage(ex)
}
})
innerSource
}
}
}
object OneToOneOnDemandSink {
def apply[T, M](sink: T => Sink[T, M]): Sink[T, NotUsed] = Sink.fromGraph(new OneToOneOnDemandSink(sink))
}
This will create a new Sink for each element so it avoids a whole lot of the complexity that LazySink
has and there is also no sensible materialized value to return. 这将为每个元素创建一个新的Sink,因此它避免了
LazySink
具有的大量复杂性,并且也没有合理的物化价值。
Below are three equivalent approaches: 以下是三种等效方法:
map
and an inner graph, map
和内部图表, flatMapConcat
and an inner graph, flatMapConcat
和内部图形, GraphDSL
. GraphDSL
再详细GraphDSL
。 In all cases, the output is: 在所有情况下,输出是:
$ tail -n +1 -- *.txt
==> 1.txt <==
1
==> 2.txt <==
2
==> 3.txt <==
3
==> 4.txt <==
4
==> 5.txt <==
5
Using map
: 使用
map
:
import java.nio.file.Paths import akka.actor.ActorSystem import akka.stream._ import akka.stream.scaladsl.{FileIO, Sink, Source} import akka.util.ByteString import scala.concurrent.Future object Example extends App { override def main(args: Array[String]): Unit = { implicit val system = ActorSystem("Example") implicit val materializer = ActorMaterializer() val result: Future[Seq[Future[IOResult]]] = Source(1 to 5) .map( elem => Source.single(ByteString(s"$elem\\n")) .runWith(FileIO.toPath(Paths.get(s"$elem.txt"))) ) .runWith(Sink.seq) implicit val ec = system.dispatcher result.onComplete(_ => system.terminate()) } }
Explanation : We map
the Int
element to a function which creates an inner graph, Source.single(ByteString(...)).runWith(FileIO.toPath(...)
, which serializes and writes to a dynamic path, and allows us to accumulate the resulting Future[IOResult]
via Sink.seq
. 说明 :我们将
Int
元素map
到一个函数,该函数创建一个内部图形, Source.single(ByteString(...)).runWith(FileIO.toPath(...)
,它序列化并写入动态路径,并允许我们通过Sink.seq
积累最终的Future[IOResult]
。
Documentation : 文件 :
map
Transform each element in the stream by calling a mapping function with it and passing the returned value downstream.map
通过调用映射函数并将返回值传递给下游来转换流中的每个元素。emits when the mapping function returns an element
映射函数返回元素时发出
backpressures when downstream backpressures
背压时下游的背压
completes when upstream completes
上游完成时完成
See also : 另见 :
Using flatMapConcat
: 使用
flatMapConcat
:
import java.nio.file.Paths import akka.actor.ActorSystem import akka.stream._ import akka.stream.scaladsl.{FileIO, Sink, Source} import akka.util.ByteString import scala.concurrent.Future object Example extends App { override def main(args: Array[String]): Unit = { implicit val system = ActorSystem("Example") implicit val materializer = ActorMaterializer() val result: Future[Seq[Future[IOResult]]] = Source(1 to 5) .flatMapConcat( elem => Source.single( Source.single(ByteString(s"$elem\\n")) .runWith(FileIO.toPath(Paths.get(s"$elem.txt"))) ) ) .runWith(Sink.seq) implicit val ec = system.dispatcher result.onComplete(_ => system.terminate()) } }
Explanation : flatMapConcat
requires a Source
. 说明 :
flatMapConcat
需要Source
。 We therefore create one which emits the mat
of the inner graph, Source.single(ByteString(...)).runWith(FileIO.toPath(...)
, which allows us to accumulate the resulting Future[IOResult]
via Sink.seq
. Actual serialization and dispatch is done by the inner graph. 因此,我们创造一个发射所述
mat
内的图形, Source.single(ByteString(...)).runWith(FileIO.toPath(...)
这使我们能够累积所得的Future[IOResult]
通过Sink.seq
。实际的序列化和调度由内部图形完成。
Documentation : 文件 :
flatMapConcat
Transform each input element into aSource
whose elements are then flattened into the output stream through concatenation.flatMapConcat
将每个输入元素转换为Source
,然后通过连接将其元素展平为输出流。 This means each source is fully consumed before consumption of the next source starts.这意味着在消耗下一个源开始之前,每个源都被完全消耗。
emits when the current consumed substream has an element available
当前消耗的子流具有可用元素时发出
backpressures when downstream backpressures
背压时下游的背压
completes when upstream completes and all consumed substreams complete
完成时上游和完成全部耗尽子完整
See also : 另见 :
Custom Sink
using the GraphDSL
: 使用
GraphDSL
自定义Sink
器:
import java.nio.file.Path import akka.stream.scaladsl.{Broadcast, FileIO, Flow, GraphDSL, Sink, Source, ZipWith} import akka.stream.{IOResult, Materializer, SinkShape} import akka.util.ByteString import scala.concurrent.Future object FileSinks { def dispatch[T]( dispatcher: T => Path, serializer: T => ByteString )( implicit materializer: Materializer ): Sink[T, Future[Seq[Future[IOResult]]]] = Sink.fromGraph( GraphDSL.create( Sink.seq[Future[IOResult]] ) { implicit builder => sink => // prepare this sink's graph elements: val broadcast = builder.add(Broadcast[T](2)) val serialize = builder.add(Flow[T].map(serializer)) val dispatch = builder.add(Flow[T].map(dispatcher)) val zipAndWrite = builder.add(ZipWith[ByteString, Path, Future[IOResult]]( (bytes, path) => Source.single(bytes).runWith(FileIO.toPath(path))) ) // connect the graph: import GraphDSL.Implicits._ broadcast.out(0) ~> serialize ~> zipAndWrite.in0 broadcast.out(1) ~> dispatch ~> zipAndWrite.in1 zipAndWrite.out ~> sink // expose ports: SinkShape(broadcast.in) } ) } ---- import java.nio.file.Paths import FileSinks import akka.actor.ActorSystem import akka.stream._ import akka.stream.scaladsl.Source import akka.util.ByteString import scala.concurrent.Future object Example extends App { override def main(args: Array[String]): Unit = { implicit val system = ActorSystem("Example") implicit val materializer = ActorMaterializer() val result: Future[Seq[Future[IOResult]]] = Source(1 to 5) .runWith(FileSinks.dispatch[Int]( elem => Paths.get(s"$elem.txt"), elem => ByteString(s"$elem\\n")) ) implicit val ec = system.dispatcher result.onComplete(_ => system.terminate()) } }
dispatcher
is a function to convert your input object to a path, this is where you can dynamically decide on the path. dispatcher
是一个将输入对象转换为路径的函数,您可以在此处动态决定路径。 serializer
is a function to serialise your input object. serializer
是一个序列化输入对象的函数。 GraphDSL
, see also: https://doc.akka.io/docs/akka/current/stream/stream-graphs.html#constructing-graphs GraphDSL
,另见: https : GraphDSL
Disclaimer : I am myself still in the process of learning Akka Stream. 免责声明 :我本人仍在学习Akka Stream。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.