[英]Flink strange “Cannot Serialize operator object class …CoBroadcastWithNonKeyedOperator” error
我正在嘗試使用BroadcastState設置項目,但是由於某些原因,當我嘗試運行該項目時出現此錯誤:
org.apache.flink.streaming.runtime.tasks.StreamTaskException:無法序列化操作員對象類org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperator。
我不確定為什么要扔它。 傳入和輸出的對象(SampleInput和Token)是非常簡單的avro生成的pojo,具有兩個或三個字段,我嘗試將BroadcastProcessFunction的方法留空以切出可以設置的任何內容,以使其無法序列化,但是仍然出現錯誤。 這是代碼的相關部分:
//Sideoutput that error strings will be written to
OutputTag<String> sideOutputTag = new OutputTag<String>("side-output") {};
//<Setup for broadcast state>
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.seconds(1))
.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
.cleanupFullSnapshot()
.build();
final MapStateDescriptor<String, Token> ruleStateDescriptor = new MapStateDescriptor<>(
"oathTokens",
BasicTypeInfo.STRING_TYPE_INFO,
AvroTypeInfo.of(new TypeHint<Token>() {}));
ruleStateDescriptor.enableTimeToLive(ttlConfig);
DataStream<Token> tokenObjectStream = tokenSourceStream.process(new JsonToTokenProcessFunction(sideOutputTag))
.startNewChain()
.uid("tokenObjectStream")
.name("tokenObjectStream");
BroadcastStream<Token> ruleBroadcastStream = tokenObjectStream.broadcast(ruleStateDescriptor);
//</Config for broadcast state>
//<Main Data Input Stream>
DataStream<SampleInput> jsonToSampleInput = kafkaStream.process(new JsonToPojoProcessFunction(sideOutputTag))
.startNewChain()
.uid("sampleInputStream")
.name("sampleInputStream");
BroadcastConnectedStream<SampleInput, Token> broadcastConnectedStream = jsonToSampleInput.connect(ruleBroadcastStream);
DataStream<SampleInput> matchedBroadcastStream = broadcastConnectedStream.process(new BroadcastProcessFunction<SampleInput, Token, SampleInput>() {
@Override
public void processElement(SampleInput sampleInput, ReadOnlyContext readOnlyContext, Collector<SampleInput> collector) throws Exception {
}
@Override
public void processBroadcastElement(Token token, Context context, Collector<SampleInput> collector) throws Exception {
}
});
任何幫助將不勝感激。 我確定我只是忽略了一些東西。 謝謝!
原來ttlConfig對象是無法序列化的。 刪除它可以解決問題。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.