如何在特定时间内运行 Spark 结构化流作业？

Question

I want to schedule a Spark structured streaming job each day. The Job itself must run for a certain number of hours and then stop. So, how can I specify such time duration?

Answer 1

You need to schedule job with databricks scheduler once a day and then in the code add a timeout to your query:

query = (df.writeStream...)

  query.awaitTermination(timeoutInSeconds)
  query.stop()