![](/img/trans.png)
[英]Exception in thread “delete Spark local dirs” java.lang.NullPointerException
[英]Why does Spark on YARN in cluster mode fail with “Exception in thread ”Driver“ java.lang.NullPointerException”?
我正在使用emr-5.4.0和Spark 2.1.0。 我理解NullPointerException
是什么,這個問題是關於為什么在這種特殊情況下拋出它。
無法弄清楚為什么我在驅動程序線程中得到NullPointerException。
我得到了這個奇怪的工作失敗了這個錯誤:
18/03/29 20:07:52 INFO ApplicationMaster: Starting the user application in a separate Thread
18/03/29 20:07:52 INFO ApplicationMaster: Waiting for spark context initialization...
Exception in thread "Driver" java.lang.NullPointerException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
18/03/29 20:07:52 ERROR ApplicationMaster: Uncaught exception:
java.lang.IllegalStateException: SparkContext is null but app is still running!
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:415)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:254)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:766)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
18/03/29 20:07:52 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.IllegalStateException: SparkContext is null but app is still running!)
18/03/29 20:07:52 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: java.lang.IllegalStateException: SparkContext is null but app is still running!)
18/03/29 20:07:52 INFO ApplicationMaster: Deleting staging directory hdfs://<ip-address>.ec2.internal:8020/user/hadoop/.sparkStaging/application_1522348295743_0010
18/03/29 20:07:52 INFO ShutdownHookManager: Shutdown hook called
End of LogType:stderr
我提交了這份工作:
spark-submit --deploy-mode cluster --master yarn --num-executors 40 --executor-cores 16 --executor-memory 100g --driver-cores 8 --driver-memory 100g --class <package.class_name> --jars <s3://s3_path/some_lib.jar> <s3://s3_path/class.jar>
我的班級看起來像這樣:
class MyClass {
def main(args: Array[String]): Unit = {
val c = new MyClass()
c.process()
}
def process(): Unit = {
val sparkConf = new SparkConf().setAppName("my-test")
val sparkSession: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate()
import sparkSession.implicits._
....
}
...
}
將class MyClass
更改為object MyClass
,您就完成了。
當我們在它的時候,我也將class MyClass
更改為object MyClass extends App
並刪除def main(args: Array[String]): Unit
(由extends App
給出)。
我已經報告了Spark 2.3.0的改進 - [SPARK-23830]當集群部署模式下的YARN上的Spark在Spark應用程序是Scala類而不是對象時失敗,並且NullPointerException - 讓它很好地報告給最終用戶。
深入研究YARN上的Spark如何工作,以下消息是當Spark應用程序的ApplicationMaster啟動驅動程序時 (您使用了--deploy-mode cluster --master yarn
with spark-submit
)。
ApplicationMaster:在單獨的線程中啟動用戶應用程序
在INFO消息之后,您應該看到另一個消息:
ApplicationMaster:等待spark上下文初始化...
這是ApplicationMaster運行時驅動程序初始化的一部分。
Exception in thread "Driver" java.lang.NullPointerException
異常Exception in thread "Driver" java.lang.NullPointerException
的原因是由於以下代碼 :
val mainMethod = userClassLoader.loadClass(args.userClass)
.getMethod("main", classOf[Array[String]])
我的理解是mainMethod
為null
,因此以下行 (其中mainMethod
為null
)“觸發” NullPointerException
:
mainMethod.invoke(null, userArgs.toArray)
該線程確實稱為Driver
(如在Exception in thread "Driver" java.lang.NullPointerException
中的Exception in thread "Driver" java.lang.NullPointerException
),如下所示 :
userThread.setContextClassLoader(userClassLoader)
userThread.setName("Driver")
userThread.start()
行號不同,因為我使用Spark 2.3.0來引用行,而使用emr-5.4.0和Spark 2.1.0。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.