Spark结构化流水印错误

Question

跟进这个问题<\/a>

我有格式与以下相同的 json 流数据

|  A    | B                                        |
|-------|------------------------------------------|
|  ABC  |  [{C:1, D:1}, {C:2, D:4}]                | 
|  XYZ  |  [{C:3, D :6}, {C:9, D:11}, {C:5, D:12}] |

Answer 1

As per my understanding, watermarking is required only when you are performing window operation on event time. Spark used watermarking to handle late data and for the same purpose Spark needs to save older aggregation.

The following link explains this very well with example: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking

I don't see any window operations in your transformation and if that is the case then I think you can try running the stream query without watermarking.

Answer 2

在对火花流结构进行分组时，您必须在数据帧中已经有了水印，并在分组时将其考虑在内，方法是在您的聚合中包含水印窗口。

    df.groupBy(col("dummy"), window(col("event_time"), "1 day")).