如何为大数据集群分配物理资源？

Question

I have three servers and I want to deploy Spark Standalone Cluster or Spark on Yarn Cluster on that servers. Now I have some questions about how to allocate physical resources for a big data cluster. For example, i want to know whether i can deploy Spark Master Process and Spark Worker Process on the same node. Why?

Server Details:

CPU Cores: 24
Memory: 128GB

I need your help. Thanks.

Answer 1

Of course you can, just put host with Master in slaves. On my test server I have such configuration, master machine is also worker node and there is one worker-only node. Everything is ok

However be aware, that is worker will fail and cause major problem (ie system restart), then you will have problem, because also master will be afected.

Edit: Some more info after question edit :) If you are using YARN (as suggested), you can use Dynamic Resource Allocation. Here are some slides about it and here article from MapR. It a very long topic how to configure memory properly for given case, I think that these resources will give you much knowledge about it

BTW. If you have already intalled Hadoop Cluster, maybe try YARN mode ;) But it's out of topic of question