遠程 flink 作業查詢 Hive 上的紗線集群錯誤：NoClassDefFoundError: org/apache/hadoop/mapred/JobConf

Question

環境：HDP：3.1.5（hadoop：3.1.1，hive：3.1.0），Flink：1.12.2 Java 代碼：

public static void main(String[] args) {
    EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().build();
    TableEnvironment tblEnv=TableEnvironment.create(settings);

    String name = "myhive";
    String defaultDatabase = "default";
    String hiveConfDir = "/etc/hive/conf";
    HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir);
    tblEnv.registerCatalog("myhive", hive);
    tblEnv.useCatalog("myhive");

    //tblEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
    tblEnv.sqlQuery("SELECT * FROM users").execute().print();
}

依賴：

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-api-java-bridge_2.12</artifactId>
  <version>${flink.version}</version>
</dependency>

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-connector-hive_2.12</artifactId>
  <version>${flink.version}</version>
</dependency>

錯誤1：

org.apache.flink.util.FlinkException: JobMaster for job 35afe414e1dd861c86130ddd031312f2 failed.
    at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:887) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
    at org.apache.flink.runtime.dispatcher.Dispatcher.dispatcherJobFailed(Dispatcher.java:465) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
    at org.apache.flink.runtime.dispatcher.Dispatcher.handleDispatcherJobResult(Dispatcher.java:444) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
    ...
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
    at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:494) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_292]
    ...
Caused by: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator Source: HiveSource-zjdev_xiangliang.users -> SinkConversionToTuple2
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:231) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
    at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:866) ~[flink-dist_2.12-1.12.2.jar:1.12.2]
    ...
Caused by: java.lang.NoClassDefFoundError: Lorg/apache/hadoop/mapred/JobConf;
    at java.lang.Class.getDeclaredFields0(Native Method) ~[?:1.8.0_292]
    at java.lang.Class.privateGetDeclaredFields(Class.java:2583) ~[?:1.8.0_292]
    at java.lang.Class.getDeclaredField(Class.java:2068) ~[?:1.8.0_292]
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1871) ~[?:1.8.0_292]
    ...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_292]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_292]
    ...

嘗試添加依賴

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-mapreduce-client-core</artifactId>
      <version>${hadoop.version}</version>
      <scope>provided</scope>
    </dependency>

得到另一個錯誤

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
    at org.apache.flink.runtime.entrypoint.parser.CommandLineOptions.<clinit>(CommandLineOptions.java:27)
    at org.apache.flink.runtime.entrypoint.DynamicParametersConfigurationParserFactory.options(DynamicParametersConfigurationParserFactory.java:43)
    at org.apache.flink.runtime.entrypoint.DynamicParametersConfigurationParserFactory.getOptions(DynamicParametersConfigurationParserFactory.java:50)
    at org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:42)
    at org.apache.flink.runtime.entrypoint.ClusterEntrypointUtils.parseParametersOrExit(ClusterEntrypointUtils.java:63)
    at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)

嘗試修復 commons-cli:1.3.1 與 1.2 的沖突：選擇 1.3.1 然后錯誤 1； 選擇 1.2 然后選擇錯誤 2； 添加依賴commons-cli 1.4，然后錯誤1。

Answer 1

1、commons-cli choose 1.3.1 or 1.4
2、add $hadoop_home/../hadoop_mapreduce/* to yarn.application.classpath

Answer 2

我也遇到了這個錯誤，當我認為我可能是版本沖突時，因為我使用的 hive 3.1.2 版本高於 hadoop 2.7.6 版本。 i always got error such as guava version conflict and so on so when i used flink-1.15.0 and link hive with jar flink-sql-connector-hive-3.1.2_2.12-1.15.0.jar i got Caused by: java.lang.ClassNotFoundException：org.apache.hadoop.mapred.JobConf
resolve: this error it because not found jars please move your hadoop jars hadoop-mapreduce-client-core.,hadoop-common,hadoop-mapreduce-client-common,hadoop-mapreduce-client-jobclient and hive-exec-3.1.2 .jar 到 flink lib 路徑