I am trying to set up a project using BroadcastState, but for some reason I am getting this error when I try to run it:
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot serialize operator object class org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperator.
I am not sure why it is throwing it. The objects passed into and output by it (SampleInput and Token) are very simple avro generated pojos with two or three fields and I tried just leaving the BroadcastProcessFunction's methods blank to cut out anything I could be setting to make in unable to be serialized, but still getting the error. Here is the relevant part of the code:
//Sideoutput that error strings will be written to
OutputTag<String> sideOutputTag = new OutputTag<String>("side-output") {};
//<Setup for broadcast state>
StateTtlConfig ttlConfig = StateTtlConfig
.newBuilder(Time.seconds(1))
.setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
.cleanupFullSnapshot()
.build();
final MapStateDescriptor<String, Token> ruleStateDescriptor = new MapStateDescriptor<>(
"oathTokens",
BasicTypeInfo.STRING_TYPE_INFO,
AvroTypeInfo.of(new TypeHint<Token>() {}));
ruleStateDescriptor.enableTimeToLive(ttlConfig);
DataStream<Token> tokenObjectStream = tokenSourceStream.process(new JsonToTokenProcessFunction(sideOutputTag))
.startNewChain()
.uid("tokenObjectStream")
.name("tokenObjectStream");
BroadcastStream<Token> ruleBroadcastStream = tokenObjectStream.broadcast(ruleStateDescriptor);
//</Config for broadcast state>
//<Main Data Input Stream>
DataStream<SampleInput> jsonToSampleInput = kafkaStream.process(new JsonToPojoProcessFunction(sideOutputTag))
.startNewChain()
.uid("sampleInputStream")
.name("sampleInputStream");
BroadcastConnectedStream<SampleInput, Token> broadcastConnectedStream = jsonToSampleInput.connect(ruleBroadcastStream);
DataStream<SampleInput> matchedBroadcastStream = broadcastConnectedStream.process(new BroadcastProcessFunction<SampleInput, Token, SampleInput>() {
@Override
public void processElement(SampleInput sampleInput, ReadOnlyContext readOnlyContext, Collector<SampleInput> collector) throws Exception {
}
@Override
public void processBroadcastElement(Token token, Context context, Collector<SampleInput> collector) throws Exception {
}
});
Any help would be greatly appreciated. I am sure I am just overlooking something. Thanks!
Turns out the ttlConfig object is what wasn't serializable. Removing it resolved the issue.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.