![](/img/trans.png)
[英]Unable to instantiate external Hive Metastore in Postgres / driver not found in classpath
[英]Exception while running hive support on Spark: Unable to instantiate SparkSession with Hive support because Hive classes are not found
您好,我正在嘗試將 Hive 與 spark 一起使用,但是當我嘗試執行時,它會顯示此錯誤
Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
這是我的源代碼
package com.spark.hiveconnect
import java.io.File
import org.apache.spark.sql.{Row, SaveMode, SparkSession}
object sourceToHIve {
case class Record(key: Int, value: String)
def main(args: Array[String]){
val warehouseLocation = new File("spark-warehouse").getAbsolutePath
val spark = SparkSession
.builder()
.appName("Spark Hive Example")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
import spark.implicits._
import spark.sql
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive")
sql("LOAD DATA LOCAL INPATH '/usr/local/spark3/examples/src/main/resources/kv1.txt' INTO TABLE src")
sql("SELECT * FROM src").show()
spark.close()
}
}
這是我的 build.sbt 文件。
name := "SparkHive"
version := "0.1"
scalaVersion := "2.12.10"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.5"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.5"
而且我還在控制台中運行了 hive。 誰能幫我這個? 謝謝你。
嘗試添加
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.5"
主要問題是 class "org.apache.hadoop.Z8A4AC216FB230DA30834DE641BZConf."D 無法加載。 您可以插入以下代碼進行測試。
Class.forName("org.apache.hadoop.hive.conf.HiveConf",true,
Thread.currentThread().getContextClassLoader);
並且在這一行會發生錯誤。
確切地說,根本問題是您的pom可能不支持Spark上的hive。 您可以檢查以下依賴項。
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.4.3</version>
</dependency>
class “org.apache.hadoop.Z8A4AC216FB230DA3834DE641B3E5D0FConfyZ.conf 位於”
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.