[英]hive execution engine - Spark - Failed to create spark client
When I issue a select query on HIVE am getting the following error 当我对HIVE发出选择查询时,出现以下错误
FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
I followed few web sites on Memory configuration but nothing really helped. 我关注了几个有关内存配置的网站,但没有任何帮助。 The settings that i have in my environment are as follows. 我在我的环境中具有的设置如下。 Can someone please help. 有人可以帮忙吗?
yarn-site.xml
yarn.nodemanager.resource.memory-mb - 4096
yarn.nodemanager.resource.cpu-vcores - 2
yarn.scheduler.minimum-allocation-mb - 128
yarn.scheduler.maximum-allocation-mb - 1024
mapred-site.xml
mapreduce.framework.name - yarn
mapreduce.map.memory.mb - 4096
mapreduce.reduce.memory.mb - 8192
mapreduce.map.java.opts - -Xmx3072m
mapreduce.reduce.java.opts - -Xmx6144m
hive-site.xml
hive.execution.engine - spark
spark.home - /.../.../binary/spark-1.6.1-bin-hadoop2.6
spark.master - yarn-client
spark.eventLog.enabled - true
spark.eventLog.dir - /.../.../binary/spark-1.6.1-bin-hadoop2.6/logs
spark.executor.memory - 512m
spark.executor.cores - 2
spark-env.sh
export SPARK_WORKER_CORES=4
export SPARK_WORKER_MEMORY=16384M
export SPARK_EXECUTOR_CORES=1
export SPARK_EXECUTOR_MEMORY=8192M
You can use hive -e in hive cli, watch the log in console 您可以在hive cli中使用hive -e,在控制台中查看日志
hive --hiveconf hive.root.logger=DEBUG,console -e "select count(1) from tableA;"
I solve the problem with change the /user dir permission on HDFS. 我通过更改HDFS上的/ user dir权限解决了问题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.