如何使用 Spark 读取 Hive 表

Question

I would like to read hive table with Spark. Hive tables data are stored as textFile in /user/hive/warehouse/problem7.db .

I do:

val warehouseLocation = hdfs://localhost:9000/user/hive/warehouse
// Create the Spark Conf and the Spark Session
val conf = new SparkConf().setAppName("Spark Hive").setMaster("local[2]").set("spark.sql.warehouse.dir", warehouseLocation)
val spark = SparkSession.builder.config(conf).enableHiveSupport().getOrCreate()

val table1 = spark.sql("select * from problem7.categories")

table1.show(false)

I have the following error:

Table or view not found: `problem7`.`categories`

Answer 1

I resolved in the following way :

I create a hive-site.xml in spark/conf and add :

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>hive.metastore.uris</name>
        <value>thrift://localhost:9083</value>
      </property>
    </configuration>

then I start hive metastore service with the following command

hive --service metastore