使用合并后如何从 Delta 表中获取新/更新的记录？

Question

在 spark 流作业中使用合并<\/a>到 Delta 表进行upsert<\/a>后，有什么方法可以获取更新\/插入的行？

val df = spark.readStream(...)
val deltaTable = DeltaTable.forName("...")


def upsertToDelta(events: DataFrame, batchId: Long) {

deltaTable.as("table")
    .merge(
      events.as("event"), 
      "event.entityId == table.entityId")
    .whenMatched()
        .updateExpr(...))
    .whenNotMatched()
      .insertAll()
    .execute()
}

df
  .writeStream
  .format("delta")
  .foreachBatch(upsertToDelta _)
  .outputMode("update")
  .start()

Answer 1

You can enabled Change Data Feed on the table, and then have another stream or batch job to fetch the changes, so you'll able to receive information on what rows has changed/deleted/inserted. It could be enabled with:

ALTER TABLE table_name SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

if thable isn't registered, you can use path instead of table name:

ALTER TABLE delta.`path` SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

The changes will be available if you add the .option("readChangeFeed", "true") option when reading stream from a table:

spark.readStream.format("delta") \
  .option("readChangeFeed", "true") \
  .table("table_name")

and it will add three columns to table describing the change - the most important is _change_type (please note that there are two different types for update operation).

If you're worried about having another stream - it's not a problem, as you can run multiple streams inside the same job - you just don't need to use .awaitTermination , but something like spark.streams.awaitAnyTermination() to wait on multiple streams.

PS But maybe this answer will change if you explain why you need to get changes inside the same job?