ClassNotFoundException: org.apache.hadoop.conf.Configuration 啟動 Flink SQL 客戶端

Question

我試圖建立蜂巢整合與弗林克如圖所示這里。 我已經按照上述配置了所有內容，所有服務（Hive、MySQL、Kafka）都在正常運行。 但是，當我使用以下命令在獨立的本地 Flink 集群上啟動 Flink SQL 客戶端時：

./bin/sql-client.sh embedded

我得到一個ClassNotFoundException: org.apache.hadoop.conf.Configuration ...

這是帶有異常跟蹤的詳細日志文件：

2020-04-01 11:27:31,458 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2020-04-01 11:27:31,459 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2020-04-01 11:27:31,459 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2020-04-01 11:27:31,460 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.process.size, 1568m
2020-04-01 11:27:31,460 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2020-04-01 11:27:31,460 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2020-04-01 11:27:31,460 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.execution.failover-strategy, region
2020-04-01 11:27:31,507 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2020-04-01 11:27:31,527 WARN  org.apache.flink.client.cli.CliFrontend                       - Could not load CLI class org.apache.flink.yarn.cli.FlinkYarnSessionCli.
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1076)
at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1030)
at org.apache.flink.table.client.gateway.local.LocalExecutor.<init>(LocalExecutor.java:135)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:85)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 7 more
2020-04-01 11:27:31,543 INFO  org.apache.flink.table.client.gateway.local.LocalExecutor     - Using default environment file: file:/home/rudip7/flink/conf/sql-client-defaults.yaml
2020-04-01 11:27:31,861 INFO  org.apache.flink.table.client.config.entries.ExecutionEntry   - Property 'execution.restart-strategy.type' not specified. Using default value: fallback
2020-04-01 11:27:32,776 ERROR org.apache.flink.table.client.SqlClient                       - SQL Client must stop. Unexpected exception. This is a bug. Please consider filing an issue.
org.apache.flink.table.client.gateway.SqlExecutionException: Could not create execution context.
at             
org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:753)
at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:228)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:98)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory.createCatalog(HiveCatalogFactory.java:84)
at org.apache.flink.table.client.gateway.local.ExecutionContext.createCatalog(ExecutionContext.java:371)
at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$null$4(ExecutionContext.java:547)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$initializeCatalogs$5(ExecutionContext.java:546)
at org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:240)
at org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:545)
at org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:494)
at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:159)
at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:118)
at org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:742)
... 3 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 14 more

你對我遺漏的東西有什么暗示嗎？ 我的類路徑上有我所有的 Hadoop 庫，但我不明白為什么缺少這個類......

我正在使用 Flink 1.10 和 Java 8

此外，日志文件中的這一行讓我懷疑這是否真的是我的錯誤：

2020-04-01 11:27:32,776 ERROR org.apache.flink.table.client.SqlClient                       - SQL Client must stop. Unexpected exception. This is a bug. Please consider filing an issue.

先感謝您！

Answer 1

如果你沒有在你的環境中添加 HADOOP_HOME，你可以在運行 ./bin/sql-client.sh 嵌入之前導出 HADOOP_CLASSPATH

export HADOOP_CLASSPATH=`hadoop classpath`