I would like to read hive table with Spark. Hive tables data are stored as textFile in /user/hive/warehouse/problem7.db
.
I do:
val warehouseLocation = hdfs://localhost:9000/user/hive/warehouse
// Create the Spark Conf and the Spark Session
val conf = new SparkConf().setAppName("Spark Hive").setMaster("local[2]").set("spark.sql.warehouse.dir", warehouseLocation)
val spark = SparkSession.builder.config(conf).enableHiveSupport().getOrCreate()
val table1 = spark.sql("select * from problem7.categories")
table1.show(false)
I have the following error:
Table or view not found: `problem7`.`categories`
I resolved in the following way :
I create a hive-site.xml in spark/conf and add :
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9083</value>
</property>
</configuration>
then I start hive metastore service with the following command
hive --service metastore
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.