當我使用spark-submit運行我的job.py時，它總是說文件'pyspark.zip'不存在

Question

環境：spark-2.1

當我使用spark-submit運行我的job.py時，總是說pyspark.zip文件不存在。

我發現了這篇文章： Spark Python提交錯誤：文件不存在：pyspark.zip ，

但它仍然無法工作。 我的run.sh如下：

#!/bin/sh
/usr/lib/software/spark/spark-2.1/bin/spark-submit \
  --master yarn-cluster \
  --driver-memory 5G \
  --num-executors 12 \
  ./job.py 

我的job.py如下：

# -*- coding:utf-8 -*-  
import sys 
import os
from pyspark.ml.feature import Word2Vec
from pyspark import SparkConf, SparkContext
from pyspark.sql import Row 
import jieba.posseg as posseg 
import jieba
if __name__ == "__main__":
    reload(sys)  
    sys.setdefaultencoding('utf-8')
    sc = SparkContext(appName="fenci_0")
    date = '20180801' 
    scatelist = ['95']
    #compute 
    sc.stop()

我運行命令：nohup ./run.sh> log 2>＆1＆。

日志文件如下：

nohup: 忽略輸入
Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
18/08/21 15:29:28 WARN util.NativeCodeLoader main: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/08/21 15:29:29 INFO yarn.Client main: Requesting a new application from cluster with 2313 NodeManagers
18/08/21 15:29:29 INFO yarn.Client main: Verifying our application has not requested more than the maximum memory capability of the cluster (26624 MB per container)
18/08/21 15:29:29 INFO yarn.Client main: Will allocate AM container, with 5632 MB memory including 512 MB overhead
18/08/21 15:29:29 INFO yarn.Client main: Setting up container launch context for our AM
18/08/21 15:29:29 INFO yarn.Client main: Setting up the launch environment for our AM container
18/08/21 15:29:29 INFO yarn.Client main: Preparing resources for our AM container
18/08/21 15:29:31 INFO yarn.Client main: Source and destination file systems are the same. Not copying viewfs://xxxx-cluster/home/spark/jar/spark-2.1.2-bin-U1/spark-2.1.2-bin-U1.zip
18/08/21 15:29:31 INFO yarn.Client main: Source and destination file systems are the same. Not copying viewfs://xxxx-cluster/home/spark/files/2.1/log4j.properties
18/08/21 15:29:31 INFO yarn.Client main: Uploading resource file:/home/job.py -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/job.py
18/08/21 15:29:32 INFO yarn.Client main: Uploading resource file:/usr/lib/software/spark/spark-2.1/python/lib/pyspark.zip -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/pyspark.zip
18/08/21 15:29:33 INFO yarn.Client main: Uploading resource file:/usr/lib/software/spark/spark-2.1/python/lib/py4j-0.10.4-src.zip -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/py4j-0.10.4-src.zip
18/08/21 15:29:33 INFO yarn.Client main: Uploading resource file:/tmp/spark-cae7c762-dc5e-4081-8b17-c1c136072780/__spark_conf__4062455229532468331.zip -> viewfs://xxxx-cluster/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/__spark_conf__.zip
18/08/21 15:29:34 INFO spark.SecurityManager main: Changing view acls to: hdp_lbg_ectech,dr.who
18/08/21 15:29:34 INFO spark.SecurityManager main: Changing modify acls to: hdp_lbg_ectech
18/08/21 15:29:34 INFO spark.SecurityManager main: Changing view acls groups to: 
18/08/21 15:29:34 INFO spark.SecurityManager main: Changing modify acls groups to: 
18/08/21 15:29:34 INFO spark.SecurityManager main: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hdp_lbg_ectech, dr.who); groups with view permissions: Set(); users  with modify permissions: Set(hdp_lbg_ectech); groups with modify permissions: Set()
18/08/21 15:29:34 INFO yarn.Client main: Submitting application application_1534831354007_17334 to ResourceManager
18/08/21 15:29:34 INFO impl.YarnClientImpl main: Submitted application application_1534831354007_17334
18/08/21 15:29:35 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)
18/08/21 15:29:35 INFO yarn.Client main: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.spark.hdp_lbg_ectech.spark
     start time: 1534836574582
     final status: UNDEFINED
     tracking URL: http://xxxxxxxxxxx:9088/proxy/application_1534831354007_17334/
     user: hdp_lbg_ectech
18/08/21 15:29:36 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)
18/08/21 15:29:37 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)
18/08/21 15:29:38 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)
18/08/21 15:29:39 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: SUBMITTED)
18/08/21 15:29:40 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:40 INFO yarn.Client main: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.spark.hdp_lbg_ectech.spark
     start time: 1534836574582
     final status: UNDEFINED
     tracking URL: http://xxxxxxxxxxx:9088/proxy/application_1534831354007_17334/
     user: hdp_lbg_ectech
18/08/21 15:29:41 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:42 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:43 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:44 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:45 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:46 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:47 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:48 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:49 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:50 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:51 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:52 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:53 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:54 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:55 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:56 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:57 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:58 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:29:59 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:00 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:01 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:02 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:03 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:04 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:05 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:06 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:07 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:08 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:09 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:10 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:11 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:12 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:13 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:14 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:15 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:16 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:17 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:18 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:19 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:20 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:21 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:22 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:23 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:24 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:25 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:26 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:27 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:28 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:29 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:30 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:31 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:32 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:33 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:34 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:35 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:36 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:37 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:38 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:39 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:40 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:41 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:42 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:43 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:44 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:45 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:46 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:47 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:48 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:49 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:50 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:51 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:52 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:53 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:54 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:55 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:56 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:57 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:58 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:30:59 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:00 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:01 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:02 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:03 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:04 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:05 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:06 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:07 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:08 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:09 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:10 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:11 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:12 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:13 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:14 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:15 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:16 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:17 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:18 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:19 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:20 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:21 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:22 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:23 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:24 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:25 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:26 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:27 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:28 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:29 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:30 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:31 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:32 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:33 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:34 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:35 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:36 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:37 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:38 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:39 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:40 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:41 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:42 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:43 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:44 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:45 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:46 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:47 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:48 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:49 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:50 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:57 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:58 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:31:59 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:00 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:02 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:03 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:04 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:05 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:06 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:07 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:08 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:09 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:10 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:11 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:12 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:13 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:14 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:15 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:16 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:17 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:18 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:19 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:20 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:21 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: ACCEPTED)
18/08/21 15:32:22 INFO yarn.Client main: Application report for application_1534831354007_17334 (state: FAILED)
18/08/21 15:32:22 INFO yarn.Client main: 
     client token: N/A
     diagnostics: Application application_1534831354007_17334 failed 4 times due to AM Container for appattempt_1534831354007_17334_000004 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://xxxxxxxxxxx:9088/proxy/application_1534831354007_17334/Then, click on links to logs of each attempt.
Diagnostics: File does not exist: hdfs://xxxx:8020/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/pyspark.zip
java.io.FileNotFoundException: File does not exist: hdfs://xxxx:8020/home/spark/stagingDir/.sparkStaging/application_1534831354007_17334/pyspark.zip
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1135)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1127)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1127)
    at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
    at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1672)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Failing this attempt. Failing the application.
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.spark.hdp_lbg_ectech.spark
     start time: 1534836574582
     final status: FAILED
     tracking URL: http://xxxxxx:9088/cluster/app/application_1534831354007_17334
     user: hdp_lbg_ectech
Exception in thread "main" org.apache.spark.SparkException: Application application_1534831354007_17334 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1185)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1231)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/08/21 15:32:22 INFO util.ShutdownHookManager Thread-3: Shutdown hook called
18/08/21 15:32:22 INFO util.ShutdownHookManager Thread-3: Deleting directory /tmp/spark-cae7c762-dc5e-4081-8b17-c1c136072780

大多數帖子都說“ pyspark.zip文件不存在”的原因是在某處將主服務器設置為local的代碼。但是我找不到在任何地方將主服務器設置為local ，在run.sh ，我設置了yarn-cluster 。 ”

您能幫我解決這個問題嗎？ 非常感謝！

Answer 1

您正在使用--master yarn-cluster ，它已棄用，如您發布的日志在第一行中所述。 這可能是造成您問題的原因，但是我不確定。 嘗試將您的Shell腳本更改為：

#!/bin/sh
/usr/lib/software/spark/spark-2.1/bin/spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --driver-memory 5G \
  --num-executors 12 \
  ./job.py 

請參閱： https ： //spark.apache.org/docs/2.1.0/submitting-applications.html

或者，您可以嘗試在客戶端模式下運行應用程序。 然后，驅動程序將在本地計算機上運行（從中運行spark-submit ），但是執行程序將在YARN工作程序節點上運行。 這是使用--deploy-mode client 。 這可能會解決您的錯誤，但是您可能有特定的原因要在YARN節點上運行驅動程序。

Answer 2

我終於找到了我的錯誤。因為日志處於運行狀態。我們應該出於一些簡單的原因只查找運行狀態。要查找失敗狀態的詳細原因，我們應該找到應用程序的標准輸出日志_ ***** **********。 最后我在代碼計算部分找到了。我使用collect（），導致內存溢出。當我刪除collect（）時，它起作用了。 很抱歉沒有顯示我的計算零件代碼。