堆棧內存溢出
  簡體   English   中英

Spark-Submit 命令需要一段時間才能運行

[英]Spark-Submit command takes a while to run

 原文  2022-05-10 18:46:28  0   1    apache-spark/ hadoop/ aix

問題描述

我們在 IBM AIX（版本 2）操作系統上運行的服務器集群上安裝了 Apache Hadoop 和 Spark。

Hadoop 版本 - hadoop-3.2.1 Spark 版本 - spark-3.0.1

我們正在通過運行$SPARK_HOME/bin文件夾下的spark-submit --version命令來測試 Spark 的整體安裝。 此命令間歇性地運行。 當您第一次運行它時，該命令會立即運行。 之后多次運行該命令需要很長時間才能執行（大約 30 - 40 分鍾）。 我們檢查了服務器 CPU 和內存 - 內存不足或應用程序占用處理器資源沒有問題。 我們無法確定此命令運行時的延遲位置。

此 Hadoop/Spark 設置在運行 Red Hat 7.9 的集群中運行。 在這種環境下，我們不會面臨這個問題。

這是我第一次在 Stack Overflow 上提問。 如果我需要提供更多信息，請告訴我。

提前致謝。

============================ 5月11日編輯：

日志運行成功（在 spark-submit 命令中添加了調試行）

bash-5.0$ spark-submit --version
Entered spark submit
About to execute spark submit command.....
About to load spark env.sh
Loaded spark env.sh
Entered statement to create RUNNER
searching spark_home/jars
Loaded spark jars DIR
Launching class path
Launched class path
Entering build command
Completed build command
About to enter while block
Entered while block for Entered build command
Entered build command
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for
For  changing delim to blank
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/java8_64/bin/java
Entered if condition for /u01/app/java8_64/bin/java
CMD is /u01/app/java8_64/bin/java
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -cp
Entered if condition for -cp
CMD is /u01/app/java8_64/bin/java -cp
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
Entered if condition for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -Xmx1g
Entered if condition for -Xmx1g
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for org.apache.spark.deploy.SparkSubmit
Entered if condition for org.apache.spark.deploy.SparkSubmit
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for --version
Entered if condition for --version
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for 0
Entered if condition for 0
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version 0
build_command is  and org.apache.spark.deploy.SparkSubmit --version
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version 0
completed while block
About to execute /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g 
   org.apache.spark.deploy.SparkSubmit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/

Using Scala version 2.12.10, IBM J9 VM, 1.8.0_251
Branch HEAD
Compiled by user ubuntu on 2020-08-28T08:58:35Z
Revision 2b147c4cd50da32fe2b4167f97c8142102a0510d
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.

==================================================== ===========

運行失敗：

bash-5.0$ spark-submit --version
Entered spark submit
About to execute spark submit command.....
About to load spark env.sh
Loaded spark env.sh
Entered statement to create RUNNER
searching spark_home/jars
Loaded spark jars DIR
Launching class path
Launched class path
Entering build command
Completed build command
About to enter while block
Entered while block for Entered build command
Entered build command
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for
For  changing delim to blank
CMD is
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/java8_64/bin/java
Entered if condition for /u01/app/java8_64/bin/java
CMD is /u01/app/java8_64/bin/java
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -cp
Entered if condition for -cp
CMD is /u01/app/java8_64/bin/java -cp
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
Entered if condition for /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for -Xmx1g
Entered if condition for -Xmx1g
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for org.apache.spark.deploy.SparkSubmit
Entered if condition for org.apache.spark.deploy.SparkSubmit
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for --version
Entered if condition for --version
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version
build_command is  and org.apache.spark.deploy.SparkSubmit --version
Entered while block for 0
Entered if condition for 0
CMD is /u01/app/java8_64/bin/java -cp /u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/conf/:/u01/app/rmb/ria/AnthemSpark/spark-3.0.1-bin-hadoop3.2/jars/*:/u01/app/rmb/ria/AnthemSpark/hadoop-3.2.1/etc/hadoop/ -Xmx1g org.apache.spark.deploy.SparkSubmit --version 0
build_command is  and org.apache.spark.deploy.SparkSubmit --version

##########################

編輯 - 3 月 12 日

這是運行以下命令時的最后幾行 - truss -d。 最后一個輸出顯示它進入“睡眠”狀態。

0.9063:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9066:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9068:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9071:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
0.9073:        kread(0, " o", 1)                = 1
0.9075:        kread(0, " r", 1)                = 1
0.9078:        kread(0, " g", 1)                = 1
0.9080:        kread(0, " .", 1)                = 1
0.9082:        kread(0, " a", 1)                = 1
0.9084:        kread(0, " p", 1)                = 1
0.9086:        kread(0, " a", 1)                = 1
0.9089:        kread(0, " c", 1)                = 1
0.9091:        kread(0, " h", 1)                = 1
0.9093:        kread(0, " e", 1)                = 1
0.9095:        kread(0, " .", 1)                = 1
0.9097:        kread(0, " s", 1)                = 1
0.9100:        kread(0, " p", 1)                = 1
0.9102:        kread(0, " a", 1)                = 1
0.9104:        kread(0, " r", 1)                = 1
0.9106:        kread(0, " k", 1)                = 1
0.9108:        kread(0, " .", 1)                = 1
0.9111:        kread(0, " d", 1)                = 1
0.9113:        kread(0, " e", 1)                = 1
0.9115:        kread(0, " p", 1)                = 1
0.9117:        kread(0, " l", 1)                = 1
0.9119:        kread(0, " o", 1)                = 1
0.9122:        kread(0, " y", 1)                = 1
0.9124:        kread(0, " .", 1)                = 1
0.9126:        kread(0, " S", 1)                = 1
0.9128:        kread(0, " p", 1)                = 1
0.9130:        kread(0, " a", 1)                = 1
0.9132:        kread(0, " r", 1)                = 1
0.9135:        kread(0, " k", 1)                = 1
0.9137:        kread(0, " S", 1)                = 1
0.9139:        kread(0, " u", 1)                = 1
0.9141:        kread(0, " b", 1)                = 1
0.9143:        kread(0, " m", 1)                = 1
0.9187:        kread(0, " i", 1)                = 1
0.9190:        kread(0, " t", 1)                = 1
0.9192:        kread(0, "\0", 1)                = 1
0.9195:        incinterval(0, 0x0FFFFFFFFFFFE5C0, 0x0FFFFFFFFFFFE5E0) = 0
0.9197:        _sigaction(14, 0x0FFFFFFFFFFFE690, 0x0FFFFFFFFFFFE6C0) = 0
0.9200:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9204:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9207:        kioctl(0, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
0.9211:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9214:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9216:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9219:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
0.9222:        kread(0, " -", 1)                = 1
0.9224:        kread(0, " -", 1)                = 1
0.9227:        kread(0, " v", 1)                = 1
0.9229:        kread(0, " e", 1)                = 1
0.9231:        kread(0, " r", 1)                = 1
0.9234:        kread(0, " s", 1)                = 1
0.9236:        kread(0, " i", 1)                = 1
0.9238:        kread(0, " o", 1)                = 1
0.9241:        kread(0, " n", 1)                = 1
0.9243:        kread(0, "\0", 1)                = 1
0.9245:        incinterval(0, 0x0FFFFFFFFFFFE5C0, 0x0FFFFFFFFFFFE5E0) = 0
0.9248:        _sigaction(14, 0x0FFFFFFFFFFFE690, 0x0FFFFFFFFFFFE6C0) = 0
0.9251:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9254:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9257:        kioctl(0, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
0.9260:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9262:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9265:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9268:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
0.9270:        kread(0, " 0", 1)                = 1
0.9273:        kread(0, "\0", 1)                = 1
0.9275:        incinterval(0, 0x0FFFFFFFFFFFE5C0, 0x0FFFFFFFFFFFE5E0) = 0
0.9278:        _sigaction(14, 0x0FFFFFFFFFFFE690, 0x0FFFFFFFFFFFE6C0) = 0
0.9281:        kfcntl(1, F_GETFL, 0x0000000000000000) = 67110914
0.9284:        kfcntl(1, F_GETFL, 0x0000000000000020) = 67110914
0.9287:        kioctl(0, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
0.9290:        lseek(0, 0, 1)                   Err#29 ESPIPE
0.9292:        fstatx(0, 0x0FFFFFFFFFFFE8F8, 176, 0) = 0
0.9295:        _sigaction(14, 0x0FFFFFFFFFFFE710, 0x0FFFFFFFFFFFE740) = 0
0.9297:        incinterval(0, 0x0FFFFFFFFFFFE640, 0x0FFFFFFFFFFFE660) = 0
2.9303:        kread(0, "\t", 1) (sleeping...)

1 個解決方案

解決方案1
 1  2022-06-21 21:18:50

@LorinczyZsigmond 的建議最終將我們引導到了正確的位置。 使用最后一行搜索 - kread(0, "\t", 1) (sleeping...) - 導致我們在最新版本中發現 Bash read builtin 問題，其中有人喋喋不休地談論其他人與發布的版本有相同問題IBM AIX Linux 工具箱網站。 更新到版本 5.1.4.2 后，問題消失了，我們能夠完成腳本。

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 無法運行“spark-submit”命令 運行 spark-submit 命令時無法將“spark”設置為服務帳戶名稱 spark-submit無法識別為內部或外部命令 找不到spark-submit -class命令？ spark-submit 命令，包括 mysql 連接器 Spark-submit命令的內存參數 如何在集群模式下運行 spark-submit 命令時覆蓋 spark jars？ （好的http3） spark-submit 命令顯示未找到 python 在氣流中找不到spark-submit命令 火花提交缺少命令行補全
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM