Spark作为Hive的引擎

Question

Can we use Spark as an engine for Hive?

We have many legacy systems and code base in Hive and would like to use Spark with Hive.

Best,

Answer 1

two options, if you want to run hive on spark, it's possible but it's a very alpha feature : https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

IMHO you're better off using Spark SQL and the designated HiveContext, that you can use like that :

// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
sqlContext.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src")

// Queries are expressed in HiveQL
sqlContext.sql("FROM src SELECT key, value").collect().foreach(println)

All in all, I'd suggest you load your Hive tables using Spark SQL and reuse all your Hive queries but through Spark SQL's APIs.

You can find more informations here : http://spark.apache.org/docs/1.2.0/sql-programming-guide.html

Does that answer your question ?

Regards,

Answer 2

You can execute hive queries on spark execution engine.Hive has a large number of dependencies, these dependencies are not included in the default Spark distribution. If Hive dependencies can be found on the classpath, Spark will load them automatically.

You can find the code here: spark-hive-example