[英]Update concurrent map inside a stream map on flink
I have one stream that constantly streaming the latest values of some keys. 我有一个流,不断地流某些键的最新值。
Stream A: DataStream[(String,Double)]
流A:
DataStream[(String,Double)]
I have another stream that wants to get the latest value on each process call. 我还有另一个要在每个流程调用中获取最新值的流。
My approach was to introduce a concurrentHashMap
which will be updated by stream A and read by the second stream. 我的方法是引入
concurrentHashMap
,它将由流A更新并由第二个流读取。
val rates = new concurrentHasMap[String,Double].asScala
val streamA : DataStream[(String,Double)]= ???
streamA.map(keyWithValue => rates(keyWithValue._1)= keyWithValue._2) //rates never gets updated
rates("testKey")=2 //this works
val streamB: DataStream[String] = ???
streamB.map(str=> rates(str) // rates does not contain the values of the streamA at this point
//some other functionality
)
Is it possible to update a concurrent map from a stream? 是否可以从流中更新并发映射? Any other solution on sharing data from a stream with another is also acceptable
任何其他与流共享数据流的解决方案也是可以接受的
The behaviour You are trying to use will not work in a distributed manner, basically if You will have parellelism
> 1 it will not work. 您尝试使用的行为将无法以分布式方式起作用,基本上,如果您的
parellelism
> 1,则该行为将不起作用。 In Your code rates
are actually updated, but in different instance of parallel operator. 在您的代码
rates
实际上是更新的,但是在并行运算符的不同实例中。
Actually, what You would like to do in this case is use a BroadcastState
which was designed to solve exactly the issue You are facing. 实际上,在这种情况下,您想要执行的操作是使用
BroadcastState
,该状态旨在完全解决您面临的问题。
In Your specific usecase it would look like something like this: 在您的特定用例中,它看起来像这样:
val streamA : DataStream[(String,Double)]= ???
val streamABroadcasted = streamA.broadcast(<Your Map State Definition>)
val streamB: DataStream[String] = ???
streamB.connect(streamABroadcasted)
Then You could easily use BroadcastProcessFunction
to implement Your logic. 然后,您可以轻松地使用
BroadcastProcessFunction
来实现您的逻辑。 More on the Broadcast state pattern can be found here 可以在这里找到有关广播状态模式的更多信息
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.