如果在類內未創建火花會話，則在紗線群集模式下，火花作業的最終狀態將失敗

Question

我遇到的一種情況是，如果不滿足某些條件，則無需在類內創建spark會話，並且應用程序會因混亂而退出。

我正在以“ yarn-cluster”模式提交以下工作

spark2-submit --class com.test.TestSpark --master yarn --deploy-mode client /home/test.jar false

作業的最終狀態為"failed" 。

但是，如果在“ yarn-client”模式下運行相同的任務，則火花作業將成功完成。

下面是代碼：

package com.test;

import org.apache.spark.sql.SparkSession;

public class TestSpark {

    public static void main(String[] args) {
        boolean condition = false;
        condition = Boolean.parseBoolean(args[0]);
        if(condition){
            SparkSession sparkSession = SparkSession.builder().appName("Data Ingestion Framework")
                    .config("hive.metastore.warehouse.dir", "/user/hive/warehouse").config("spark.sql.warehouse.dir", "/user/hive/warehouse")
                    .enableHiveSupport()
                    .getOrCreate();
        }else{
        System.out.println("coming out no processing required");
        }

    }

}

在“ yarn-cluster ”的日志中，我可以看到正在創建兩個着色器，其中一個失敗，並出現以下錯誤：

18/05/09 18:21:51 WARN security.UserGroupInformation: PriviledgedActionException as:*****<uername> (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://hostname/user/*****<uername>/.sparkStaging/application_1525778267559_0054/__spark_conf__.zip
Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://hostname/user/*****<uername>/.sparkStaging/application_1525778267559_0054/__spark_conf__.zip
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1257)
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1249)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1249)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$4$$anonfun$apply$3.apply(ApplicationMaster.scala:198)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$4$$anonfun$apply$3.apply(ApplicationMaster.scala:195)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$4.apply(ApplicationMaster.scala:195)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$4.apply(ApplicationMaster.scala:160)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:787)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)

您能否解釋一下為什么會發生這種情況以及spark如何處理容器的創建。

Answer 1

阿米特，這是一個尚待解決的已知問題。 https://issues.apache.org/jira/browse/SPARK-10795

解決方法是初始化SparkContext。

package com.test;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SparkSession;

public class TestSpark {

    public static void main(String[] args) {
        boolean condition = false;
        condition = Boolean.parseBoolean(args[0]);
        if(condition){
            SparkSession sparkSession = SparkSession.builder().appName("Data Ingestion Framework")
                    .config("hive.metastore.warehouse.dir", "/user/hive/warehouse").config("spark.sql.warehouse.dir", "/user/hive/warehouse")
                    .enableHiveSupport()
                    .getOrCreate();
        }else{
            // Initialize a spark context to avoid failure : https://issues.apache.org/jira/browse/SPARK-10795
            JavaSparkContext sparkContext = new JavaSparkContext(new SparkConf());
            System.out.println("coming out no processing required");
        }

    }