Flink作业：获取InvalidClassException：com.fasterxml.jackson.databind.ObjectMapper

Question

Our flink job is running perfectly fine in local but while trying to deploy it in yarn cluster it is throwing below error .

Flink version : 1.5.0 Jackson 2.6.7

Caused by: java.io.InvalidClassException: com.fasterxml.jackson.databind.ObjectMapper; local class incompatible: stream classdesc serialVersionUID = 2, local class serialVersionUID = 1

I have tried with different flink versions and the issue is coming with all the versions.

DataStream stream = DataStreamManager
            .getStatusUpdateStream(kafkaConsumerConfig, env)
            .filter(new CheckStatusUpdate())
            .keyBy(new OrderUpdateKeySelector());

    stream.process(new ProcessBookingUpdatesEvents(config)).setParallelism(processBookingUpdatesEventsParallelism)
            .process(new ExecuteOrderStatusUpdateNotifierFilter(config)).setParallelism(filterParallelism)
            .process(new ProcessOrderPostFiltering(config,communicationConfig)).setParallelism(processOrderPostFilteringParallelism);

Command to submit flink job:

export HADOOP_CLASSPATH=`hadoop classpath`;HADOOP_CONF_DIR=/etc/hadoop/conf flink-1.5.0/bin/flink run -m yarn-cluster -yn 2 -ys 1  -yjm 2048 -ytm 4096  -yst -ynm  order_updates -yqu default -p 1 -c com.order.workflow.consumer.flink.OrderStatusUpdateJob  orchestrator-build-1.0.21-staging.jar  config.yml

Answer 1

Welcome to the classpath-hell :-). With the information you provide I can't really see where this error appears but I guess somewhere within a flink-operator you try to use an ObjectMapper to marshal/unmarshal some data. I would recommend you to check what (which jackson version) is on your yarn classpath. If you need a specific version of jackson build a fat-jar with a shaded jackson version. If not play the easy game and take whether the one which is already your yarn classpath or (maybe even easier) take the one flink already shaded for you: You can find under the package name: org.apache.flink.hadoop.shaded.org.codehaus.jackson.map .

Hope that helps.

Answer 2

Jackson version coming from yarn was 2.9.5 and the one i was using was 2.6.7 Changing the jackson version to 2.9.5 worked for me.

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.9.5</version>
    </dependency>