如何获取增量表的最新插入时间？

Question

In my Spark structured streaming application, I have a code like this.

df = (
  spark.readStream.format("delta")
  .option("startingTimestamp", starting_time_stamp)
  .table(t)
)

Now if the given starting timestamp is later than the timestamp on which the last insertion was done, I get an error. So, my question is how can I check for the latest timestamp on which an insertion was committed?

Answer 1

我们可以通过以下代码找到最新的时间戳。

ts = spark.sql("SELECT max(timestamp) FROM (DESCRIBE HISTORY <table>)").first()[0]