如何处理 Kubernetes 上 Spark-submit 的错误代码 101

Question

我正在尝试运行以下代码以将 Spark 应用程序提交到 Kubernetes 的集群：

 /opt/spark/bin/spark-submit --master k8s://https://<spark-master-ip>:6443 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=newfrontdocker/spark:v3.0.1-j14 local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar 100

当我尝试运行此代码时，容器会在我最终收到此特定错误消息并且容器终止之前暂时运行：

 container status:
                 container name: spark-kubernetes-driver
                 container image: newfrontdocker/spark:v3.0.1-j14
                 container state: terminated
                 container started at: 2021-07-17T11:49:46Z
                 container finished at: 2021-07-17T11:49:48Z
                 exit code: 101
                 termination reason: Error
21/07/17 06:49:13 INFO LoggingPodStatusWatcherImpl: Application status for spark-c15c11340f204794b51bf8d79397bf9e (phase: Failed)
21/07/17 06:49:13 INFO LoggingPodStatusWatcherImpl: Container final statuses:


         container name: spark-kubernetes-driver
         container image: newfrontdocker/spark:v3.0.1-j14
         container state: terminated
         container started at: 2021-07-17T11:49:46Z
         container finished at: 2021-07-17T11:49:48Z
         exit code: 101
         termination reason: Error
21/07/17 06:49:13 INFO LoggingPodStatusWatcherImpl: Application spark-pi with submission ID default:spark-pi-4ed9627ab44c778d-driver finished
21/07/17 06:49:14 INFO ShutdownHookManager: Shutdown hook called
21/07/17 06:49:14 INFO ShutdownHookManager: Deleting directory /tmp/spark-3d1a7ff3-1dc9-4db0-acfa-ed52706122b6

退出代码 101 是什么？我需要做些什么来解决这个问题，以便在 kubernetes 的集群上运行 spark 应用程序？

Answer 1

就我而言，原因是 jar 路径不正确。

我想路径应该是：

local:///opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar

（3.0.1 而不是 3.1.1），因为您使用newfrontdocker/spark:v3.0.1-j14映像。