When I try to write my own apache beam demo by using kafka to pub/sub data from shopping system, using beam to design the data flow, run on flink. I stuck on a very rare exception:
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.flink.streaming.api.operators.InternalTimer, but class was expected at org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator.fireTimer(WindowDoFnOperator.java:129) at org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.onProcessingTime(DoFnOperator.java:704) at org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:235) at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:285)
my code is:
package com.meikeland.dataflow;
import org.apache.beam.runners.flink.FlinkRunner;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.kafka.KafkaIO;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.*;
import org.apache.beam.sdk.transforms.windowing.*;
import org.apache.beam.sdk.values.KV;
import org.apache.kafka.common.serialization.LongDeserializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class GameStats {
private static final Logger logger = LoggerFactory.getLogger(GameStats.class);
public static void main(String[] args) {
KFOptions options = PipelineOptionsFactory.fromArgs(args).as(KFOptions.class);
options.setRunner(FlinkRunner.class);
options.setStreaming(true);
logger.info("brokers address is: {}", options.getBrokers());
runDemoCount(options);
}
private static void runDemoCount(KFOptions options) {
Pipeline pipeline = Pipeline.create(options);
pipeline
// read order events from kafka
.apply("ConsumeKafka",
KafkaIO.<Long, String>read().withBootstrapServers(options.getBrokers()).withTopic("tracking.order.goods")
.withKeyDeserializer(LongDeserializer.class).withValueDeserializer(StringDeserializer.class)
.withLogAppendTime().withoutMetadata())
.apply(Values.create()).apply("ParseOrderInfo", ParDo.of(new ParseOrderInfoFn()))
.apply("SetTimestamp", WithTimestamps.of(OrderInfo::getCreatedAt))
.apply("ExtractOrderID", MapElements.via(new SimpleFunction<OrderInfo, Integer>() {
public Integer apply(OrderInfo o) {
logger.info("processed orderID: {}", o.getOrderID());
return o.getOrderID();
}
}))
// window
.apply("FixedWindowsOrderID",
Window.<Integer>into(FixedWindows.of(new Duration(1000 * 60)))
.triggering(AfterWatermark.pastEndOfWindow()
.withEarlyFirings(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(new Duration(1000 * 60)))
.withLateFirings(AfterPane.elementCountAtLeast(1)))
.withAllowedLateness(new Duration(1000 * 60)).accumulatingFiredPanes())
.apply("Count", Count.<Integer>perElement()).apply("ToString", ParDo.of(new DoFn<KV<Integer, Long>, String>() {
@ProcessElement
public void processElement(@Element KV<Integer, Long> element, IntervalWindow window,
OutputReceiver<String> r) {
logger.info("the order is : {}, and count is : {}", element.getKey(), element.getValue());
r.output(String.format("interval :%s, Order ID: %d, Count :%d", window.start().toString(), element.getKey(),
element.getValue()));
}
})).apply("WriteToKafka", KafkaIO.<Void, String>write().withBootstrapServers(options.getBrokers())
.withTopic("streaming.order.count").withValueSerializer(StringSerializer.class).values());
pipeline.run().waitUntilFinish();
}
}
It seems the error is in the window, but I can't figure it out. And I google everywhere, no one seems came across similar error. So I must be make some little thing wrong. Please who can save me.
I had the same issue and I fixed it by checking if the version of flink is compatible with Beam:
https://beam.apache.org/documentation/runners/flink/
In my case I have Beam 2.6 and flink 1.5.4.
I hope it will help you.
Regards, Ali
I also have this problem and finally figure it out.
If your project depends on
"org.apache.beam" % "beam-runners-flink" % beamVersion
which uses the InternalTimer Class
I take a look of scala API document of org.apache.flink.streaming
and the InternalTimer
become Interface
after Flink 1.6
.
In order to properly use Apache Beam FlinkRunner
with InternalTimer Interface
after Flink 1.6
, your project has to depend on
"org.apache.beam" % "beam-runners-flink-1.6" % beamVersion
or
"org.apache.beam" % "beam-runners-flink-1.7" % beamVersion
or
"org.apache.beam" % "beam-runners-flink-1.8" % beamVersion
Everything would be great
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.