在Spark Streaming中处理太晚的数据

Question

Watermark allows late arriving data to be considered for inclusion against already computed results for a period of time using windows. Its premise is that it tracks to a point in time before which it is assumed no more late events are supposed to arrive, but if they do, they are none-the-less discarded .

Is there a way to store the discarded data, that can be used for reconciliation purpose later? Say In my Structured Streaming, I set the watermark to 1 hour. I am doing window operation for each 10 min and received a later event 20 min late. Is there a way I can store the discarded data say at a different location rather than discarding it?

Answer 1

不，没有办法实现这一方面。