在spark上配置配置单元的问题

Question

I have downloaded spark-2.0.0-bin-hadoop2.7. Could any one advise how to configure hive on this and use in scala console? Now I am able to run RDD's on file using Scala (spark-shell console).

Answer 1

将您的hive-site.xml放在spark conf目录中

Answer 2

Hive can support multiple execution engine. Like TEZ, Spark. You can set the property in hive-site.xml

</property> 
<name>hive.execution.engine</name>
<value>spark</value>
<description>
 I am choosing Spark as the execution engine
</description>
</property>

Copy jars spark-assembly jar to HIVE_HOME/lib

Set the spark_home

set the below properties

set spark.master=<Spark Master URL>
set spark.eventLog.enabled=true;
set spark.eventLog.dir=<Spark event log folder (must exist)>
set spark.executor.memory=512m;             
set spark.serializer=org.apache.spark.serializer.KryoSerializer;

Above steps would suffice i think

Answer 3

Follow the official Hive on Spark documentation:

https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

You can set on Hive the spark engine by using the following command:

set hive.execution.engine=spark;

or by adding it on hive-site.xml (refer to kanishka post)

Then prior to Hive 2.2.0, copy the spark-assembly jar to HIVE_HOME/lib.

Since Hive 2.2.0, Hive on Spark runs with Spark 2.0.0 and above, which doesn't have an assembly jar.

To run with YARN mode (either yarn-client or yarn-cluster), copy the following jars to HIVE_HOME/lib.

scala-library

spark-core

spark-network-common

Set the spark_home:

export $SPARK_HOME=/path-to-spark

Start Spark Master and Workers:

spark-class org.apache.spark.deploy.master.Master

spark-class org.apache.spark.deploy.worker.Worker spark://MASTER_IP:PORT

Configure Spark:

set spark.master=<Spark Master URL>;
set spark.executor.memory=512m; 
set spark.yarn.executor.memoryOverhead=10~20% of spark.executor.memory(value);     
set spark.serializer=org.apache.spark.serializer.KryoSerializer;