Spark 2与Hive MetaStore的连接

Question

FOr the last 3 weeks, I am trying to connect to the hive metaStore remotely from my machine.

I have all the configuration files :

• hive-site
• and the configuration to the hdfs

I have already managed to use files from the hdfs, so it works.

I think I have all the jars for the spark->hive connection

the code I wrote is the following one :

import org.apache.spark.sql.Row

import org.apache.spark.sql.SparkSession

val warehouseLocation = "/user/hive/warehouse"


val spark = SparkSession
  .builder()
  .appName("SparkHiveExample")
  .config("spark.sql.warehouse.dir", warehouseLocation)
  .enableHiveSupport()
  .getOrCreate()

It throws this exception :

Unable to instantiate SparkSession with Hive support because hive classes are not found. at org.apache.sql.SparkSession$builder.enableHiveSupport

what jar am I missing ?

Observations

If I don't use enablehiveSupport(), then it works.

But then I get the next exception,

could not initialize class org.apach.spark.rdd.RDDOperationScope

Answer 1

I am not sure, but this is happening may be because you forget to export HIVE_HOME on Hive installation. So SparkSession is unable to find where to look for hive classes. As a result you also need to edit your bash_profile.

nano ~/.bash_profile

Add the following lines to your bash_profile

export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin

Save this file and then

source ~/.bash_profile

After this try to run your code. May this can solve your problem.

Answer 2

I had done this in the past and it was not very straightforward unfortunately. I created a custom distribution of spark with hive using the command

./make-distribution.sh --name my-spark-dist --tgz  -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn 

I used the client config for Hive hive-site.xml, core-site.xml and hdfs-site.xml pointing to the remote hive and hdfs and had to change firewall config to allow connection to the thrift server port.

Spark is compiled with Hive 1.2.1 and the documentation says you can use a metastore of a lower version but that doesn't work. The lowest version that works is 1.2.0 because at runtime it picks up the jars specified in your config property but at build-time it uses hive version 1.2.1 I had raised a spark bug for the same https://issues.apache.org/jira/browse/SPARK-14492 I had to upgrade my metastore DB and service upto version 1.2.0 using the upgrade tool provided with Hive