简体繁体 English

预安装火花纱簇

[英]Preinstallation for spark yarn cluster

原文 2016-03-29 02:18:03 3 2 apache-spark/ hdfs/ yarn

I tried to make a spark cluster with yarn. 我试图用纱线制作火花簇。 Do I need to install yarn to all nodes? 我需要在所有节点上安装yarn吗？ I know yarn will ship all need jar and MR but to take them, I presume that worker nodes need yarn package. 我知道纱线将运送所有需要的罐子和MR，但要拿走它们，我想工人节点需要纱线包装。 master node should be installed with HDFS and jar as well as yarn. 主节点应安装HDFS和jar以及yarn。

2 个解决方案

YARN requires two processes YARN需要两个过程

Resource Manager 资源经理
Node Manager 节点管理器

Resource Manager is the master which delegates tasks. 资源管理器是委派任务的主体。 Node Manager is the slave which works on the given piece of work. 节点管理器是负责给定工作的从属服务器。 You have to install Resource Manager on one machine(Production grade server) and Node Manager on all the slave machines(commodity hardware). 您必须在一台计算机（生产级服务器）上安装资源管理器，并在所有从属计算机（商品硬件）上安装节点管理器。

HDFS requires the following processes HDFS需要以下过程

NameNode 名称节点
DataNode 数据节点
SecondaryName SecondaryName

NameNode and Secondary NameNode are to be installed in two separate machines(Production grade servers) and DataNode on all slave machines(commodity hardware). NameNode和Secondary NameNode将安装在所有从属计算机（商品硬件）的两台单独的计算机（生产级服务器）和DataNode中。

Typically DataNode and NodeManager would be installed together on all the slave nodes. 通常，DataNode和NodeManager将一起安装在所有从属节点上。

Not sure, what you are trying to do exactly. 不确定，您到底想做什么。

Since hadoop 2.0 yarn is an integral part of hadoop. 由于hadoop 2.0纱线是hadoop不可或缺的一部分。 So, if you install hadoop, yarn automatically gets installed. 因此，如果您安装hadoop，yarn会自动安装。

When you use provided scripts to start hadoop, they will start hadoop stack. 使用提供的脚本启动hadoop时，它们将启动hadoop堆栈。 Then you can use provided script to start yarn. 然后，您可以使用提供的脚本来启动yarn。

Then you install spark, and point it to use libraries from hadoop installation, and configuration. 然后安装spark，并指向spark以使用hadoop安装和配置中的库。

There is no need to get into messy detail of installing yarn manually. 无需深入了解手动安装纱线的细节。