当不再检查状态数据时，Spark Structured Streaming如何刷新内存状态？

Question

I am trying to build a sessionization application with Spark Structured Streaming(version 2.2.0) .

In case of using mapGroupWithState with Update mode, I understand that the executor will crash with an OOM exception if the state data grows large. Hence, I have to manage the memory with GroupStateTimeout option. (Ref. How does Spark Structured Streaming handle in-memory state when state data is growing? )

However, I can't check if the state is timed-out and ready to be removed if there is no more new streaming data for the particular keys.

For example, let's say I have the following code.

myDataset
  .groupByKey(_.key)
  .flatMapGroupsWithState(OutputMode.Update, GroupStateTimeout.EventTimeTimeout)(makeSession)

makeSession() function will check if the state is timed-out and remove the timed-out state.

Now, let's say the key "foo" has some stored state in memory already, and no new data with the key "foo" is streaming into the application. As a result, makeSession() does not process the data with key "foo" and the stored state is not being checked. Which means, the stored state with key "foo" persists in memory. If there are many keys like "foo", the stored states will not be flushed and JVM will raise OOM exception.

I might be misunderstanding with mapGroupWithState , but I suspect my OOM exception is caused by the above issue.

If I am correct, what would be the solution for this case? I want to flush all the stored states that has been timedout and have no more new streaming data.

Is there any good code example?

Answer 1

Now, let's say the key "foo" has some stored state in memory already, and no new data with the key "foo" is streaming into the application. As a result, makeSession() does not process the data with key "foo" and the stored state is not being checked.

This is incorrect. As long as you have new data for any key , Spark will make sure that each batch validates the entire key set, and invoke the timed out keys one last time.

A part of every call to flat/mapGroupsWithState , we have:

val outputIterator =
      updater.updateStateForKeysWithData(filteredIter) ++
      updater.updateStateForTimedOutKeys()

And this is updateStateForTimedOutKeys :

def updateStateForTimedOutKeys(): Iterator[InternalRow] = {
  if (isTimeoutEnabled) {
    val timeoutThreshold = timeoutConf match {
      case ProcessingTimeTimeout => batchTimestampMs.get
      case EventTimeTimeout => eventTimeWatermark.get
      case _ =>
        throw new IllegalStateException(
          s"Cannot filter timed out keys for $timeoutConf")
    }
    val timingOutKeys = store.filter { case (_, stateRow) =>
      val timeoutTimestamp = getTimeoutTimestamp(stateRow)
      timeoutTimestamp != NO_TIMESTAMP && timeoutTimestamp < timeoutThreshold
    }
    timingOutKeys.flatMap { case (keyRow, stateRow) =>
      callFunctionAndUpdateState(keyRow, Iterator.empty, Some(stateRow), hasTimedOut = true)
    }
  } else Iterator.empty
}

Where the relevant part is flatMap over the timed out keys and invoking each function one last time with hasTimedOut = true .