Apache Livy 0.7.0 無法創建交互式會話

Question

使用 apache Livy 0.7.0 創建新會話時，出現以下錯誤。 我還使用 zeppelin notebook（livy interpreter）來創建會話。

使用 Scala 版本 2.12.10，Java HotSpot(TM) 64 位服務器 VM，11.0.11
火花 3.0.2 齊柏林飛艇 0.9.0

知道為什么我收到錯誤嗎？

我已經檢查過我們在類路徑中有 livy-repl_2.11-0.7.1-incubating.jar 並且 JAR 已經有它無法找到的類。

$ jar tf livy-repl_2.11-0.7.1-incubating.jar | grep "ReplDriver"
org/apache/livy/repl/ReplDriver$$anonfun$addFile$1.class
org/apache/livy/repl/ReplDriver$$anonfun$1.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$1.class
org/apache/livy/repl/ReplDriver$$anonfun$addJarOrPyFile$1.class
org/apache/livy/repl/ReplDriver.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$2.class
org/apache/livy/repl/ReplDriver$$anonfun$2.class

錯誤日志

    21/06/17 13:13:28 INFO LineBufferedStream: Exception in thread "main" java.lang.ClassNotFoundException: org.apache.livy.repl.ReplDriver
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)


21/06/17 13:13:28 WARN ContextLauncher: Child process exited with code 1.
21/06/17 13:13:28 ERROR RSCClient: Failed to connect to context.
java.io.IOException: Child process exited with code 1.
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:397)
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:448)
        at java.base/java.lang.Thread.run(Thread.java:834)
21/06/17 13:13:28 ERROR SparkProcApp: spark-submit exited with code 1
21/06/17 13:13:28 INFO RSCClient: Failing pending job f05c2bac-6e0a-4732-a419-0bde4dcfd06b due to shutdown.
21/06/17 13:13:28 INFO InteractiveSession: Stopping InteractiveSession 0...
21/06/17 13:13:28 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
21/06/17 13:13:28 INFO InteractiveSession: Stopped InteractiveSession 0.
21/06/17 13:13:28 WARN InteractiveSession: Fail to get rsc uri
java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411) 

Answer 1

Spark 3.0.x 附帶了 Scala 2.12 版本。 您需要使用 scal 2.12 使用 Spark 3.0.x 構建 livy 來解決此問題。 然后你需要調整你的 livy.conf 這里是關於如何使用 maven 重建你的 livy 的文章（ How to rebuild apache Livy with scala 2.12 ）

祝你好運。 我遇到了同樣的問題，並且能夠通過上述步驟解決。