使用kafka + beam + flink时。 找到了接口org.apache.flink.streaming.api.operators.InternalTimer，但是期望使用类

Question

When I try to write my own apache beam demo by using kafka to pub/sub data from shopping system, using beam to design the data flow, run on flink. I stuck on a very rare exception:

Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.flink.streaming.api.operators.InternalTimer, but class was expected at org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator.fireTimer(WindowDoFnOperator.java:129) at org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.onProcessingTime(DoFnOperator.java:704) at org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:235) at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:285)

my code is:

    package com.meikeland.dataflow;

    import org.apache.beam.runners.flink.FlinkRunner;
    import org.apache.beam.sdk.Pipeline;
    import org.apache.beam.sdk.io.kafka.KafkaIO;
    import org.apache.beam.sdk.options.PipelineOptionsFactory;
    import org.apache.beam.sdk.transforms.*;
    import org.apache.beam.sdk.transforms.windowing.*;
    import org.apache.beam.sdk.values.KV;
    import org.apache.kafka.common.serialization.LongDeserializer;
    import org.apache.kafka.common.serialization.StringDeserializer;
    import org.apache.kafka.common.serialization.StringSerializer;
    import org.joda.time.Duration;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;

    public class GameStats {

      private static final Logger logger = LoggerFactory.getLogger(GameStats.class);

      public static void main(String[] args) {
        KFOptions options = PipelineOptionsFactory.fromArgs(args).as(KFOptions.class);
        options.setRunner(FlinkRunner.class);
        options.setStreaming(true);
        logger.info("brokers address is: {}", options.getBrokers());
        runDemoCount(options);
      }

      private static void runDemoCount(KFOptions options) {
        Pipeline pipeline = Pipeline.create(options);

        pipeline
            // read order events from kafka
            .apply("ConsumeKafka",
                KafkaIO.<Long, String>read().withBootstrapServers(options.getBrokers()).withTopic("tracking.order.goods")
                    .withKeyDeserializer(LongDeserializer.class).withValueDeserializer(StringDeserializer.class)
                    .withLogAppendTime().withoutMetadata())
            .apply(Values.create()).apply("ParseOrderInfo", ParDo.of(new ParseOrderInfoFn()))
            .apply("SetTimestamp", WithTimestamps.of(OrderInfo::getCreatedAt))
            .apply("ExtractOrderID", MapElements.via(new SimpleFunction<OrderInfo, Integer>() {
              public Integer apply(OrderInfo o) {
                logger.info("processed orderID: {}", o.getOrderID());
                return o.getOrderID();
              }
            }))
            // window
            .apply("FixedWindowsOrderID",
                Window.<Integer>into(FixedWindows.of(new Duration(1000 * 60)))
                    .triggering(AfterWatermark.pastEndOfWindow()
                        .withEarlyFirings(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(new Duration(1000 * 60)))
                        .withLateFirings(AfterPane.elementCountAtLeast(1)))
                    .withAllowedLateness(new Duration(1000 * 60)).accumulatingFiredPanes())
            .apply("Count", Count.<Integer>perElement()).apply("ToString", ParDo.of(new DoFn<KV<Integer, Long>, String>() {
              @ProcessElement
              public void processElement(@Element KV<Integer, Long> element, IntervalWindow window,
                  OutputReceiver<String> r) {
                logger.info("the order is : {}, and count is : {}", element.getKey(), element.getValue());
                r.output(String.format("interval :%s, Order ID: %d, Count :%d", window.start().toString(), element.getKey(),
                    element.getValue()));
              }
            })).apply("WriteToKafka", KafkaIO.<Void, String>write().withBootstrapServers(options.getBrokers())
                .withTopic("streaming.order.count").withValueSerializer(StringSerializer.class).values());

        pipeline.run().waitUntilFinish();
      }
    }

It seems the error is in the window, but I can't figure it out. And I google everywhere, no one seems came across similar error. So I must be make some little thing wrong. Please who can save me.

Answer 1

I had the same issue and I fixed it by checking if the version of flink is compatible with Beam:

https://beam.apache.org/documentation/runners/flink/

In my case I have Beam 2.6 and flink 1.5.4.

I hope it will help you.

Regards, Ali

Answer 2

I also have this problem and finally figure it out.

If your project depends on

"org.apache.beam" % "beam-runners-flink" % beamVersion

which uses the InternalTimer Class

I take a look of scala API document of org.apache.flink.streaming and the InternalTimer become Interface after Flink 1.6 .

In order to properly use Apache Beam FlinkRunner with InternalTimer Interface after Flink 1.6 , your project has to depend on

"org.apache.beam" % "beam-runners-flink-1.6" % beamVersion

or

"org.apache.beam" % "beam-runners-flink-1.7" % beamVersion

or

"org.apache.beam" % "beam-runners-flink-1.8" % beamVersion

Everything would be great