使用 Spark Structured Streaming 对超时传感器数据进行分组

Question

We have sensors starting and running for a random duration multiple times a day. The data from the sensors are sent to a Kafka topic and is consumed by Spark Structured streaming API and is stored to a Delta Lake. Now we have to identify and store sessions for each sensor in a different Delta Lake table partitioned by device_id and sensor_id.

I tried with Spark Structured Streaming with watermarking but didn't do much good.

stream2 = spark.readStream.format('delta')
             .load('<FIRST_DELTA_LAKE_TABLE>')
             .select('device_id', 'json', 'time')
             .withWatermark('timestamp', '10 minutes')
             .groupBy('device_id').agg(F.min('time').alias('min_time'), F.max('time').alias('max_time')))
             .writeStream
             .format("delta")
stream2.start("<SESSIONS_TABLE>")

The idea was to have second table identifying the sessions from incoming data and saving the start time and end time for each session and device. The streaming jobs runs and nothing gets written to the Sessions delta table.

Any help on this will be appreciated.

Answer 1

By default, the when you're writing a stream, it uses the append mode by default (see the doc ). And in this mode, when you're using watermarks, the data output only after the watermark is crossed, so there will be at least 10 minutes delay until you start to see the data in the output.

But I think that the primary problem is that you're running the "global" aggregations, without defined window, or something like. Usually for session detection people use flatMapGroupWithState , something like described in the following blog post .