Dynamic Resource allocation in Spark-Yarn Cluster Mode

Question

When i use the below setting to start the spark application (default is yarn-client mode) works fine

spark_memory_setting="--master yarn --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.yarn.queue=ciqhigh --conf spark.dynamicAllocation.initialExecutors=50 --conf spark.dynamicAllocation.maxExecutors=50 --executor-memory 2G --driver-memory 4G"

ISSUE

Whereas when i change the deploy mode as cluster,application not starting up. Not even throwing any error to move on.

 spark_memory_setting="--master yarn-cluster --deploy-mode=cluster --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.yarn.queue=ciqhigh --conf spark.dynamicAllocation.initialExecutors=50 --conf spark.dynamicAllocation.maxExecutors=50 --executor-memory 2G --driver-memory 4G"

LOG

18/01/08 01:21:00 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode

This is the last line from the logger.

Any suggestions most welcome.

One important think to highlight here, the spark application which am trying to deploy starts the apache thrift server. After my searching, i think its coz of thrift am not able to able to run yarn in cluster mode. Any help to run in cluster mode.

Answer 1

选项--master yarn-cluster是错误的。.这不是有效的主url，应该只是“ yarn”而不是“ yarn-cluster” ..只是交叉检查。