Spark 無法使用內置連接提供程序連接到數據庫

Question

我正在嘗試連接到 Postgres 遵循此文檔

並且文件說內置連接提供程序。 誰能幫我解決這個問題，好嗎？ ` 有以下數據庫的內置連接提供程序：

• DB2
• 瑪麗亞數據庫
• 微軟 SQL
• 甲骨文
• PostgreSQL `

val spark = SparkSession.builder().appName("get-from-postgres").master("local[*]")
      .getOrCreate()
val jdbcDF = spark.read
  .format("jdbc")
  .option("url", url)
  .option("dbtable", table)
  .option("user", username)
  .option("password", password)
  .load()
  jdbcDF.show(10)

運行我的應用程序時，我總是收到此錯誤。

[error] java.sql.SQLException: No suitable driver
[error]         at java.sql/java.sql.DriverManager.getDriver(DriverManager.java:298)
[error]         at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$2(JDBCOptions.scala:107)
[error]         at scala.Option.getOrElse(Option.scala:201)
[error]         at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:107)
[error]         at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:39)
[error]         at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:34)
[error]         at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
[error]         at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
[error]         at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
[error]         at scala.Option.getOrElse(Option.scala:201)
[error]         at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
[error]         at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
[error]         at udw.uni.vn.loader.TESTLoader$.delayedEndpoint$udw$uni$vn$loader$TESTLoader$1(TESTLoader.scala:51)
[error]         at udw.uni.vn.loader.TESTLoader$delayedInit$body.apply(TESTLoader.scala:12)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1(App.scala:76)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:76)
[error]         at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
[error]         at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
[error]         at scala.App.main(App.scala:76)
[error]         at scala.App.main$(App.scala:74)
[error]         at udw.uni.vn.loader.TESTLoader$.main(TESTLoader.scala:12)
[error]         at udw.uni.vn.loader.TESTLoader.main(TESTLoader.scala)
[error]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error] stack trace is suppressed; run last Compile / run for the full output
[error] (Compile / run) java.sql.SQLException: No suitable driver
[error] Total time: 15 s, completed Jun 27, 2022, 4:49:01 PM

Answer 1

最后，我找到了解決方案。 僅為內置連接器添加依賴項到build.sbt

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "3.3.0",
    "org.apache.spark" %% "spark-sql" % "3.3.0",
    "joda-time" % "joda-time" % "2.10.14",
    "com.typesafe" % "config" % "1.4.1",
    // "org.mongodb.spark" % "mongo-spark-connector" % "10.0.2",
    "com.microsoft.sqlserver" % "mssql-jdbc" % "10.2.1.jre11",
    "mysql" % "mysql-connector-java" % "8.0.29",
    "org.postgresql" % "postgresql" % "42.4.0",
    "com.oracle.database.jdbc" % "ojdbc8" % "21.6.0.0.1",
    "org.mariadb.jdbc" % "mariadb-java-client" % "3.0.5",
)