简体   繁体   English

Spark-从本地计算机添加工作程序(独立的Spark集群管理器)?

[英]Spark - Add Worker from Local Machine (standalone spark cluster manager)?

When running spark 1.4.0 in a single machine, I can add worker by using this command "./bin/spark-class org.apache.spark.deploy.worker.Worker myhostname:7077". 在单台计算机上运行spark 1.4.0时,我可以使用以下命令“ ./bin/spark-class org.apache.spark.deploy.worker.Worker myhostname:7077”添加worker。 The official documentation points out another way by adding "myhostname:7077" to the "conf/slaves" file followed by executing the command "sbin/start-all.sh" which invoke the master and all workers listed in conf/slaves file. 官方文档指出了另一种方法,在“ conf / slaves”文件中添加“ myhostname:7077”,然后执行命令“ sbin / start-all.sh”,该命令将调用conf / slaves文件中列出的master和所有worker。 However, the later method doesn't work for me (with time-out error). 但是,后一种方法对我不起作用(存在超时错误)。 Can anyone help me with this? 谁能帮我这个?

Here is my conf/slaves file (assume the master URL is myhostname:700): 这是我的conf / slaves文件(假设主URL是myhostname:700):

myhostname:700 myhostname:700

conf.slaves文件应该只是主机名列表,您不需要包括spark所在的端口号(我认为如果这样做,它将尝试在该端口上使用ssh,这可能是超时的来源) )。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 为什么Spark不使用本地计算机上的所有核心 - Why is Spark not using all cores on local machine Apache spark:你能在独立模式下启动一个拥有更多内核或内存的工作者吗? - Apache spark:Can you start a worker in standalone mode with more cores or memory than physically available? Apache Spark:如何在Spark独立模式下完成应用程序级调度? - Apache Spark: How application level scheduling is done in spark standalone mode? Spark作业未在本地并行化(使用本地文件系统中的Parquet + Avro) - Spark job not parallelising locally (using Parquet + Avro from local filesystem) 在所有 Spark 工作节点上初始化 gensim 对象 - Initializing gensim objects on all spark worker nodes 如何在Spark群集中分配任务? - How are tasks distributed within a Spark cluster? Apache Spark:并行化多机器学习算法 - Apache Spark: Parallelization of Multiple Machine Learning ALgorithm 如何将特定数据分发到spark中的每个集群节点? - How to distribute specific data to each cluster node in spark? R中的并行或降雪程序包可以与火花群集交互吗? - Can the parallel or snow packages in R interface with a spark cluster? 如果数据适合单个机器,那么使用Spark是否有意义? - If data fits on a single machine does it make sense to use Spark?
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM