How to compute difference between timestamps with PySpark Structured Streaming

Question

I have the following problem with PySpark Structured Streaming.

Every line in my stream data has a user ID and a timestamp. Now, for every line and for every user, I want to add a column with the difference of the timestamps.

For example, suppose the first line that I receive says: "User A, 08:00:00". If the second line says "User A, 08:00:10" then I want to add a column in the second line called "Interval" saying "10 seconds".

Is there anyone who knows how to achieve this? I tried to use the window functions examples of the Structured Streaming documentation but it was useless.

Thank you very much

Answer 1

Since we're speaking about Structured Streaming and "every line and for every user" that tells me that you should use a streaming query with some sort of streaming aggregation ( groupBy and groupByKey ).

For streaming aggregation you can only rely on micro-batch stream execution in Structured Streaming. That gives that records for a single user could be part of two different micro-batches. That gives that you need a state.

That all together gives that you need a stateful streaming aggregation.

With that, I think you want one of the Arbitrary Stateful Operations , ie KeyValueGroupedDataset.mapGroupsWithState or KeyValueGroupedDataset.flatMapGroupsWithState (see KeyValueGroupedDataset ):

Many usecases require more advanced stateful operations than aggregations. For example, in many usecases, you have to track sessions from data streams of events. For doing such sessionization, you will have to save arbitrary types of data as state, and perform arbitrary operations on the state using the data stream events in every trigger.

Since Spark 2.2, this can be done using the operation mapGroupsWithState and the more powerful operation flatMapGroupsWithState . Both operations allow you to apply user-defined code on grouped Datasets to update user-defined state.

A state would be per user with the last record found. That looks doable.

My concerns would be:

1. How many users is this streaming query going to deal with? (the more the bigger the state)

2. When to clean up the state (of users that are no longer expected in a stream)? (which would keep the state of a reasonable size)