[英]Can't apply() custom functions to a WindowedStream on Flink
我试图为Window的apply()方法编写自定义逻辑。 基本上我想减少Window中的所有元素,然后为该值附加时间戳,所以我从DataStream创建了一个WindowedStream,但是当我尝试为apply()定义函数时,它在编译时失败了。
这是代码:
class WindowReduceFunction extends ReduceFunction[(Int, String, Int)] {
override def reduce(a: (Int, String, Int), b: (Int, String, Int)) : (Int, String, Int) = {
(a._1, a._2, a._3 + b._3)
}
}
class WindowTimestampAddFunction extends WindowFunction[(Int, String, Int), (Int, String, Int, Long), (Int, String), TimeWindow] {
override def apply(key : (Int, String), window : Window, in: Iterable[(Int, String, Int)], out: Collector[(Int, String, Int, Long)]) {
for(row <- in) {
out.collect((row._1, row._2, row._3, window.maxTimestamp()))
}
}
}
DataStream的类型为[Int,String,Int],键为[Int,String]。 没有apply()的代码运行和编译没有错误,但是当我键入时:
myWindowedStream.apply(new WindowReduceFunction(), new WindowTimestampAddFunction())
当它失败并且无法编译时,给出错误:
[ERROR] [R](preAggregator: ((Int, String, Int), (Int, String, Int)) => (Int, String, Int), windowFunction: (org.apache.flink.api.java.tuple.Tuple, org.apache.flink.streaming.api.windowing.windows.TimeWindow, Iterable[(Int, String, Int)], org.apache.flink.util.Collector[R]) => Unit)(implicit evidence$6: org.apache.flink.api.common.typeinfo.TypeInformation[R])org.apache.flink.streaming.api.scala.DataStream[R] <and>
[ERROR] [R](preAggregator: org.apache.flink.api.common.functions.ReduceFunction[(Int, String, Int)], function: org.apache.flink.streaming.api.scala.function.WindowFunction[(Int, String, Int),R,org.apache.flink.api.java.tuple.Tuple,org.apache.flink.streaming.api.windowing.windows.TimeWindow])(implicit evidence$5: org.apache.flink.api.common.typeinfo.TypeInformation[R])org.apache.flink.streaming.api.scala.DataStream[R]
[ERROR] cannot be applied to (WindowReduceFunction, WindowTimestampAddFunction)
[ERROR] .apply(new WindowReduceFunction(), new WindowTimestampAddFunction())
[ERROR] ^
[ERROR] one error found
您正在使用索引位置键,如keyBy(1)
或字段表达式键,如keyBy("field")
。 这意味着WindowedStream
的键类型是Tuple
类型( org.apache.flink.api.java.tuple.Tuple
是特定的)。
如果你将WindowFunction
的第三个泛型参数从(Int, String)
WindowFunction
为Tuple
(Int, String)
它应该可以工作。 您还可以更改keyBy
调用以使用lambda函数,然后您可以在WindowedStream
获取正确的特定键类型。 例如: keyBy( in => (in._1, in._2)
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.