通过Spark创建的Hive表在HUE / Hive GUI中不可见

Question

I am creating a hive table from scala using the next code:

  val spark = SparkSession
    .builder()
    .appName("self service")
    .enableHiveSupport()
    .master("local")
    .getOrCreate()

  spark.sql("CREATE TABLE default.TEST_TABLE (C1 INT)")

The table must be successfully created, because if I run this code twice I receive an error saying the table already exists.

However, when I try to access this table from the GUI (HUE), I cannot see any table in Hive, so it seems it's being saved in a different path that the used by Hive in HUE to get this information.

Do you know what should I do to see the tables I create from my code from the HUE/Hive web GUI?

Any help will be very appreciated. Thank you very much.

Answer 1

I seems to me you have not added hive-site.xml to the proper path. Hive-site has the properties that spark need to connect successfully with Hive and you should add this to the directory

SPARK_HOME/conf/

You can also add this file by using spark.driver.extraClassPath and give the directory where this file exists. For example in pyspark submit

/usr/bin/spark2-submit \
--conf spark.driver.extraClassPath=/../ Directory with Hive-site.xml / \
--master yarn --deploy-mode client --driver-memory nG --executor-memory nG \  
--executor-cores n myScript.py