在 Spark 中使用 jdbc 驱动程序连接到 Hive

Question

I need to move data from remote Hive to local Hive with Spark. I try to connect to remote hive with JDBC driver: 'org.apache.hive.jdbc.HiveDriver'. I'm now trying to read from Hive and the result is the column headers in the column values in stead of the actual data:

df = self.spark_session.read.format('JDBC') \
         .option('url', "jdbc:hive2://{self.host}:{self.port}/{self.database}") \
         .option('driver', 'org.apache.hive.jdbc.HiveDriver') \
         .option("user", self.username) \
         .option("password", self.password)
         .option('dbtable', 'test_table') \
         .load()
df.show()

Result:

+----------+
|str_column|
+----------+
|str_column|
|str_column|
|str_column|
|str_column|
|str_column|
+----------+

I know that Hive JDBC isn't an official support in Apache Spark. But I have already found solutions to download from other unsupported sources, such as IMB Informix. Maybe someone has already solved this problem.

Answer 1

After debug&trace the code we will find the problem in JdbcDialect。There is no HiveDialect so spark will use default JdbcDialect.quoteIdentifier。 So you should implement a HiveDialect to fix this problem:

import org.apache.spark.sql.jdbc.JdbcDialect

class HiveDialect extends JdbcDialect{
  override def canHandle(url: String): Boolean = 
    url.startsWith("jdbc:hive2")
  

  override def quoteIdentifier(colName: String): String = {
    if(colName.contains(".")){
      var colName1 = colName.substring(colName.indexOf(".") + 1)
      return s"`$colName1`"
    }
    s"`$colName`"
  }
}

And then register the Dialect by:

JdbcDialects.registerDialect(new HiveDialect)

At last, add option hive.resultset.use.unique.column.names=false to the url like this

option("url", "jdbc:hive2://bigdata01:10000?hive.resultset.use.unique.column.names=false")

refer to csdn blog

Answer 2

Apache Kyuubi has provided a Hive dialect plugin here. https://kyuubi.readthedocs.io/en/latest/extensions/engines/spark/jdbc-dialect.html

Hive Dialect plugin aims to provide Hive Dialect support to Spark's JDBC source. It will auto registered to Spark and applied to JDBC sources with url prefix of jdbc:hive2:// or jdbc:kyuubi:// . It will quote identifier in Hive SQL style, eg. Quote table.column in table . column .

1. compile and get the dialect plugin from Kyuubi. (It's a standalone Spark plugin, which is independent from Kyuubi)
2. put jar into $SPARK_HOME/jars
3. add plugin to config spark.sql.extensions=org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension , it will be auto registered to spark