如何使用 SPARK 在 HIVE 上进行查询？

Question

I am trying to use spark to run queries on hive table. I have followed lots of articles present on internet, but had no success. I have moved the hive-site.xml file to spark location.

Could you please explain how to do that? I am using Spark 1.6

Thank you in advance.

Please find my code below.

import sqlContext.implicits._
import org.apache.spark.sql
val eBayText = sc.textFile("/user/cloudera/spark/servicesDemo.csv")
val hospitalDataText = sc.textFile("/user/cloudera/spark/servicesDemo.csv")
val header = hospitalDataText.first()
val hospitalData = hospitalDataText.filter(a=>a!=header)
case class Services(uhid:String,locationid:String,doctorid:String)
val hData = hospitalData.map(_.split(",")).map(p=>Services(p(0),p(1),p(2)))
val hosService = hData.toDF()
hosService.write.format("parquet").mode(org.apache.spark.sql.SaveMode.Append).save("/user/hive/warehouse/hosdata")

This code created 'hosdata' folder at specified path, which contains data in 'parquet' format.

But when i went to hive and check table got created or not the, i did not able to see any table name as 'hosdata'.

So i run below commands.

hosService.write.mode("overwrite").saveAsTable("hosData")
sqlContext.sql("show tables").show

shows me below result

+--------------------+-----------+
|           tableName|isTemporary|
+--------------------+-----------+
|             hosdata|      false|
+--------------------+-----------+

But again when i check in hive, i can not see table 'hosdata'

Could anyone let me know what step i am missing?

Answer 1

There are multiple ways you can use to query Hive using Spark.

1. Like in Hive CLI, you can query using Spark SQL
2. Spark-shell is available to run spark class files in which you need to define variable like for hive, spark configuration object. Spark Context-sql() method allows you to execute the same query that you might have executed on Hive

Performance tuning is definitely an important perspect as you can use broadcast and other methods for faster execution.

Hope this helps.