Pipeline in Spark Structured Streaming using foreachBatch

Question

Can we chain multiple foreachBatch in Spark Structured streaming?

Eg:

df.writeStream.format("parquet")
    .foreachBatch(pre_process)
    .foreachBatch(mask)
    .trigger(once=True)
    .outputMode("append")
    .option("checkpointLocation", "location")
    .start()

If not possible, can the function in foreachBatch call multiple other batch functions?

Answer 1

Wording of question a little awkward compared to example, but...

• No chaining evident if you run two such statements.
• Two foreachBatch es run independently.
• If you look at the manual and the examples you will see that it is about writing and write guarantees.