如何使用 Databricks 的 Apache Spark 从 SQL 表中获取 stream 数据

Question

I am attempting to stream from sql table using the following:

my_sales =  spark.read.jdbc(jdbcUrl, dbo.table)

static = spark.read.format("csv").load(my_sales)
dataSchema = static.schema

I am trying to read in the data from the table with the following:

rawdf = (spark.readStream 
      .format("csv") \
      .option("maxFilesPerTrigger", 1) \
      .schema(dataSchema) \
      .csv(dataPath)
           )

I am using the following to write the data to the following location

saveloc = '/mnt/raw/streaminglocation/'


streamingQuery = (
  rawdf
  .writeStream
  .format("csv")
  .outputMode("append")
  .option("checkpointLocation", f"{saveloc}/_checkpoints")
  .option("mergeSchema", "true")
  .start(saveloc)
)

However this failing.

Is it possible to stream from a SQL table?

Answer 1

This is not possible. JDBC sources are not supported for Spark Structured Streaming.

Not convinced of the upfront coding either.

Use CDC with Kafka, or materialized updateable views with CDC with KAFKA, or Debezium.