在Spark上进行配置时，在Spark Master Web UI中的作业应用程序中总是错误的executor_cores

Question

I am trying to switch hive 2.1.1 on mapreduce to hive on spark. As told in hive on spark official site, i build a spark 1.6.0 (as to spark rev in hive 2.1.1 source code POM) without hive. The Spark is working fine with a spark- submit/spark-shell test. I set the

spark.executor.cores/spark.executor.memory

in hive-site.xml , also limit these 2 by

SPARK_WORKER_CORES/SPARK_WORKER_MEMORY

in spark-env.sh . But after i start a hive query like select count(*) from hive cli, the job in spark master web UI is always with 0 CPU cores applied, so the job is not executed and hive query waits like for ever in cli. And spark cluster is set up on a docker environment on that each server is a docker container running on a server with added up to 160 cores/160g memory . Before i set SPARK_WORKER_CORES/SPARK_WORKER_MEMORY , always 156 cores are applied which also leads to failure without enough resource. After i set SPARK_WORKER_CORES/SPARK_WORKER_MEMORY limitted to resource assigned to the docker container, 0 is applied.

i have been stuck on the problem 2 days without progress. hope some tips from anyone who is familiar with hive on docker or run hive/spark on a docker env.

Answer 1

I dont think spark execution engine works well with hive at all. The hive version you are trying integrate with spark , is built with spark 2.0.0 and not 1.6.0 There has been lot of discussion on this before . See the thread here You are better off with using Tez as many of of users reports on that thread.