无法在本地运行 PySpark（Kafka 到 Delta）并获取 SparkException：找不到目录“spark_catalog”的目录插件 class

Question

我正在尝试编写一个 PySpark 代码来读取 Kafka 主题并发布到 Delta 表。 但它不起作用。

示例代码：

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *
from delta import *

spark = SparkSession \
        .builder \
        .appName("test") \
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
        .config("spark.jars.packages", "io.delta:delta-core_2.12:2.1.0") \
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
        .getOrCreate()

kafka_df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "demo.topic") \
    .option("startingOffsets", "latest") \
    .load() \
    .withColumn("current_timestamp", unix_timestamp()) \
    .withColumn("value_str", col("value").cast(StringType())) \
    .select("current_timestamp", "value_str")

stream = kafka_df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "./data/tmp/delta/events/_checkpoints/") \
    .toTable("events")

stream.awaitTermination()

命令：星火版本：3.3.1

spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0 kafka_to_delta.py

安慰：

    Traceback (most recent call last):
      File "/Users/user/Desktop/python-module/kafka_to_delta.py", line 24, in <module>
        stream = kafka_df.writeStream \
      File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/streaming.py", line 1468, in toTable
      File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
      File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
      File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    py4j.protocol.Py4JJavaError: An error occurred while calling o63.toTable.
    : org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog
        at org.apache.spark.sql.errors.QueryExecutionErrors$.catalogPluginClassNotFoundForCatalogError(QueryExecutionErrors.scala:1638)
        at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:65)
        at org.apache.spark.sql.connector.catalog.CatalogManager.loadV2SessionCatalog(CatalogManager.scala:67)
        at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$v2SessionCatalog$2(CatalogManager.scala:86)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
        at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$v2SessionCatalog$1(CatalogManager.scala:86)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.sql.connector.catalog.CatalogManager.v2SessionCatalog(CatalogManager.scala:85)
        at org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:51)
        at org.apache.spark.sql.connector.catalog.CatalogManager.currentCatalog(CatalogManager.scala:122)
        at org.apache.spark.sql.connector.catalog.LookupCatalog.currentCatalog(LookupCatalog.scala:34)
        at org.apache.spark.sql.connector.catalog.LookupCatalog.currentCatalog$(LookupCatalog.scala:34)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.currentCatalog(Analyzer.scala:187)
        at org.apache.spark.sql.connector.catalog.LookupCatalog$CatalogAndIdentifier$.unapply(LookupCatalog.scala:111)
        at
:
:
    Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.delta.catalog.DeltaCatalog
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:55)
        ... 25 more
    
    23/01/29 18:00:53 INFO SparkContext: Invoking stop() from shutdown hook

在运行此代码之前，我是否需要指定目录和模式？ 这样做的最佳做法是什么？

Answer 1

尝试将io.delta:delta-core_2.12:2.1.0添加到通过--packages传递的包列表中（您还需要确保使用匹配版本的spark-sql-kafka ）：

spark-submit --packages \
  org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.1,io.delta:delta-core_2.12:2.1.0 \
  kafka_to_delta.py

或删除--packages并在脚本中指定 Kafka 依赖项，尽管最好不要在代码中包含特定版本，而是在其外部提供所有选项。