堆栈内存溢出
  简体   繁体   English

Spark：基于Hadoop多节点用Spark执行python脚本

[英]Spark : Execute python script with Spark based on Hadoop Multinode

 原文  2018-03-15 09:21:02  7   2    hadoop/ apache-spark/ pyspark/ cluster-computing

问题描述

I'm looking for use Spark based on Hadoop Multinodes and I have a question about my pythonic script with cluster mode.我正在寻找基于Hadoop Multinodes的Spark使用，我对我的集群模式 pythonic 脚本有疑问。

My Configuration:我的配置：

I have into my Hadoop Cluster:我进入了我的 Hadoop 集群：

  • 1 Namenode (master) 1个名称节点（主节点）
  • 2 Datanodes (slaves) 2个数据节点（奴隶）

So I would like to execute my script in Python in order to use this cluster.所以我想在 Python 中执行我的脚本以使用这个集群。 I know that Spark could be used as standalone mode, but I would like to use my nodes.我知道 Spark 可以用作独立模式，但我想使用我的节点。

My python script:我的蟒蛇脚本：

It's a very simple script which let to count words in my text.这是一个非常简单的脚本，可以用来计算文本中的单词数。

import sys
from pyspark import SparkContext

sc = SparkContext()
lines = sc.textFile(sys.argv[1])
words = lines.flatMap(lambda line: line.split(' '))
words_with_1 = words.map(lambda word: (word, 1))
word_counts = words_with_1.reduceByKey(lambda count1, count2: count1 + count2)
result = word_counts.collect()

for (word, count) in result:
    print word.encode("utf8"), count

My Spark command:我的火花命令：

In order to use Spark, I do:为了使用 Spark，我这样做：

time ./bin/spark-submit --master spark://master:7077 /home/hduser/count.py /data.txt

But, this command lets to execute Spark in standalone mode right?但是，这个命令可以在独立模式下执行 Spark，对吧？ How I can execute Spark using my Hadoop Cluster (eg yarn) and make parallel and distributed computing on my cluster?我如何使用我的 Hadoop 集群（例如 yarn）执行 Spark 并在我的集群上进行并行和分布式计算？

I tried:我试过：

time ./bin/spark-submit --master yarn /home/hduser/count.py /data.txt

And I get issues:我遇到了问题：

2018-03-15 10:13:14 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-03-15 10:13:15 INFO  SparkContext:54 - Running Spark version 2.3.0
2018-03-15 10:13:15 INFO  SparkContext:54 - Submitted application: count.py
2018-03-15 10:13:15 INFO  SecurityManager:54 - Changing view acls to: hduser
2018-03-15 10:13:15 INFO  SecurityManager:54 - Changing modify acls to: hduser
2018-03-15 10:13:15 INFO  SecurityManager:54 - Changing view acls groups to:
2018-03-15 10:13:15 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-03-15 10:13:15 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser)$
2018-03-15 10:13:16 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 40388.
2018-03-15 10:13:16 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-03-15 10:13:16 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-03-15 10:13:16 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-03-15 10:13:16 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-03-15 10:13:16 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-b131528e-849e-4ba7-94fe-c552572f12fc
2018-03-15 10:13:16 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-03-15 10:13:16 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-03-15 10:13:17 INFO  log:192 - Logging initialized @5400ms
2018-03-15 10:13:17 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2018-03-15 10:13:17 INFO  Server:414 - Started @5667ms
2018-03-15 10:13:17 INFO  AbstractConnector:278 - Started ServerConnector@4f835332{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-03-15 10:13:17 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2f867b0c{/jobs,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a0105b7{/jobs/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3fd04590{/jobs/job,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2637750b{/jobs/job/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@439f0c7{/stages,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3978d915{/stages/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596dc76d{/stages/stage,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7054d173{/stages/stage/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47b526bb{/stages/pool,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7896fc75{/stages/pool/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2fd54632{/storage,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@79dcd5f2{/storage/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1732b48c{/storage/rdd,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5888874b{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5de9bebe{/environment,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@428593b4{/environment/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4011c9bc{/executors,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5cbfbc2a{/executors/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4c33f54d{/executors/threadDump,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22c5d74c{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6cd7b681{/static,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ee342f2{/,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4d68a347{/api,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e878af1{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@590aa379{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-03-15 10:13:17 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-03-15 10:13:19 INFO  RMProxy:98 - Connecting to ResourceManager at master/172.30.10.64:8050
2018-03-15 10:13:20 INFO  Client:54 - Requesting a new application from cluster with 3 NodeManagers
2018-03-15 10:13:20 INFO  Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-03-15 10:13:20 INFO  Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2018-03-15 10:13:20 INFO  Client:54 - Setting up container launch context for our AM
2018-03-15 10:13:20 INFO  Client:54 - Setting up the launch environment for our AM container
2018-03-15 10:13:20 INFO  Client:54 - Preparing resources for our AM container
2018-03-15 10:13:24 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-03-15 10:13:29 INFO  Client:54 - Uploading resource file:/tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606/__spark_libs__580552500091841387.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/__s$
2018-03-15 10:13:33 INFO  Client:54 - Uploading resource file:/usr/local/spark/python/lib/pyspark.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/pyspark.zip
2018-03-15 10:13:33 INFO  Client:54 - Uploading resource file:/usr/local/spark/python/lib/py4j-0.10.6-src.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/py4j-0.10.6-src.zip
2018-03-15 10:13:34 INFO  Client:54 - Uploading resource file:/tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606/__spark_conf__7840630163677580304.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/__$
2018-03-15 10:13:34 INFO  SecurityManager:54 - Changing view acls to: hduser
2018-03-15 10:13:34 INFO  SecurityManager:54 - Changing modify acls to: hduser
2018-03-15 10:13:34 INFO  SecurityManager:54 - Changing view acls groups to:
2018-03-15 10:13:34 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-03-15 10:13:34 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser)$
2018-03-15 10:13:34 INFO  Client:54 - Submitting application application_1521023754917_0007 to ResourceManager
2018-03-15 10:13:34 INFO  YarnClientImpl:251 - Submitted application application_1521023754917_0007
2018-03-15 10:13:34 INFO  SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1521023754917_0007 and attemptId None
2018-03-15 10:13:35 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:35 INFO  Client:54 -
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1521105214408
         final status: UNDEFINED
         tracking URL: http://master:8088/proxy/application_1521023754917_0007/
         user: hduser
2018-03-15 10:13:36 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:37 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:38 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:39 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:40 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:41 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:42 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:43 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:44 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:45 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:46 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:47 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:48 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:49 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:50 INFO  Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:51 INFO  Client:54 - Application report for application_1521023754917_0007 (state: FAILED)
2018-03-15 10:13:51 INFO  Client:54 -
         client token: N/A
         diagnostics: Application application_1521023754917_0007 failed 2 times due to AM Container for appattempt_1521023754917_0007_000002 exited with  exitCode: -103
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1521023754917_0007Then, click on links to logs of each attempt.
Diagnostics: Container [pid=9363,containerID=container_1521023754917_0007_02_000001] is running beyond virtual memory limits. Current usage: 147.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing cont$
Dump of the process-tree for container_1521023754917_0007_02_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 9369 9363 9363 9363 (java) 454 16 2250776576 37073 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/co$
        |- 9363 9361 9363 9363 (bash) 0 0 12869632 742 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0$

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1521105214408
         final status: FAILED
         tracking URL: http://master:8088/cluster/app/application_1521023754917_0007
         user: hduser
2018-03-15 10:13:51 INFO  Client:54 - Deleted staging directory hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007
2018-03-15 10:13:51 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:214)
        at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:13:51 INFO  AbstractConnector:318 - Stopped Spark@4f835332{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-03-15 10:13:51 INFO  SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-03-15 10:13:51 WARN  YarnSchedulerBackend$YarnSchedulerEndpoint:66 - Attempted to request executors before the AM has registered!
2018-03-15 10:13:51 INFO  YarnClientSchedulerBackend:54 - Shutting down all executors
2018-03-15 10:13:51 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down
2018-03-15 10:13:51 INFO  SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
2018-03-15 10:13:51 INFO  YarnClientSchedulerBackend:54 - Stopped
2018-03-15 10:13:51 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-03-15 10:13:51 INFO  MemoryStore:54 - MemoryStore cleared
2018-03-15 10:13:51 INFO  BlockManager:54 - BlockManager stopped
2018-03-15 10:13:51 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2018-03-15 10:13:51 WARN  MetricsSystem:66 - Stopping a MetricsSystem that is not running
2018-03-15 10:13:51 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-03-15 10:13:52 INFO  SparkContext:54 - Successfully stopped SparkContext
Traceback (most recent call last):
  File "/home/hduser/count.py", line 4, in <module>
    sc = SparkContext()
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 180, in _do_init
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 270, in _initialize_context
  File "/usr/local/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1428, in __call__
  File "/usr/local/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:214)
        at java.lang.Thread.run(Thread.java:748)

2018-03-15 10:13:52 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-03-15 10:13:52 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606
2018-03-15 10:13:52 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-f5d31d54-e456-4fcb-bf48-9f950233ad4b

I'm getting all the time FAILED when I want to use my cluster with Spark当我想将我的集群与 Spark 一起使用时，我总是FAILED

在此处输入图像描述

Finally I tried:最后我尝试了：

time ./bin/spark-submit --master yarn --deploy-mode cluster /home/hduser/count.py /data.txt

But I get one more time issues.但是我又遇到了一个问题。

I don't understand something?我有什么不明白？ I'm very new with Big Data so it's possible:/ EDIT:我对大数据很陌生，所以有可能：/编辑：

This is what I obtain with: yarn application -status application_1521023754917_0007这是我获得的： yarn application -status application_1521023754917_0007

18/03/15 10:52:07 INFO client.RMProxy: Connecting to ResourceManager at master/172.30.10.64:8050
18/03/15 10:52:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Application Report : 
    Application-Id : application_1521023754917_0007
    Application-Name : count.py
    Application-Type : SPARK
    User : hduser
    Queue : default
    Start-Time : 1521105214408
    Finish-Time : 1521105231067
    Progress : 0%
    State : FAILED
    Final-State : FAILED
    Tracking-URL : http://master:8088/cluster/app/application_1521023754917_0007
    RPC Port : -1
    AM Host : N/A
    Aggregate Resource Allocation : 16329 MB-seconds, 15 vcore-seconds
    Diagnostics : Application application_1521023754917_0007 failed 2 times due to AM Container for appattempt_1521023754917_0007_000002 exited with  exitCode: -103
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1521023754917_0007Then, click on links to logs of each attempt.
Diagnostics: Container [pid=9363,containerID=container_1521023754917_0007_02_000001] is running beyond virtual memory limits. Current usage: 147.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1521023754917_0007_02_000001 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 9369 9363 9363 9363 (java) 454 16 2250776576 37073 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/tmp -Dspark.yarn.app.container.log.dir=/usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg master:40388 --properties-file /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/__spark_conf__/__spark_conf__.properties 
    |- 9363 9361 9363 9363 (bash) 0 0 12869632 742 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/tmp -Dspark.yarn.app.container.log.dir=/usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 'master:40388' --properties-file /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/__spark_conf__/__spark_conf__.properties 1> /usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001/stdout 2> /usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001/stderr 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.

2 个解决方案

解决方案1
 1  2018-03-15 10:53:47

for me this spark submit run the python on all spark nodes:对我来说，这个 spark 提交在所有 spark 节点上运行 python：

spark-submit --master yarn 
             --deploy-mode cluster 
             --num-executors 1 
             --driver-memory 2g 
             --executor-memory 1g 
             --executor-cores 1 
             hdfs://<host>:<port>/home/hduser/count.py /data.txt

The Spark environment need to be extended with: export PYSPARK_PYTHON=/opt/bin/python需要扩展 Spark 环境：export PYSPARK_PYTHON=/opt/bin/python

Furthermore the py file need to be located on hdfs so that all spark nodes in the cluster can read it.此外，py 文件需要位于 hdfs 上，以便集群中的所有 spark 节点都可以读取它。 The py file need to be accessible for the spark user. spark 用户需要可以访问 py 文件。

解决方案2
 0  2020-01-07 10:36:58

you have to first locate the py scrip in the HDFS location.您必须首先在 HDFS 位置找到 py 脚本。 With the name node correct URL使用名称节点正确的 URL

like hdfs dfs -ls hdfs://hostname:1543/像 hdfs dfs -ls hdfs://hostname:1543/

if you see your file is reflecting in the screen then this is the correct path.如果您看到您的文件反映在屏幕上，那么这是正确的路径。

Next execute接下来执行

/bin/spark-submit --master yarn hdfs://COMPLETEHOSTNAME:1543/count.py /data.txt /bin/spark-submit --master 纱线 hdfs://COMPLETEHOSTNAME:1543/count.py /data.txt

it will surely work.它肯定会起作用。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 转换Python脚本以使其能够在Spark / Hadoop中运行 - Converting Python script to be able to run in Spark/Hadoop 如何在spark中使用transform python udf执行hql脚本？ - How to execute hql script with transform python udf in spark? 如何为Spark，Python设置特定的Hadoop版本 - How to set specific Hadoop version for Spark, Python 用于Python（PySpark）的Spark的自定义Hadoop配置？ - Custom Hadoop Configuration for Spark from Python (PySpark)? 创建步骤Spark Python，亚马逊hadoop - create step spark python, amazon hadoop 无法提交Spark Python脚本 - unable to submit spark python script 在hadoop上安装spark - Installing spark on hadoop 在Spark中转储hadoop配置 - Dump hadoop configuration in Spark 更改 Spark 的 Hadoop 版本 - Change Hadoop version for Spark spark + hadoop数据位置 - spark + hadoop data locality
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM