Spark结构化流mapGroupWithState输出到实木复合地板

Question

I have a Spark Structured Streaming application which doing event merges with mapGroupWithState . It works perfect with console sink, but in production I need to write data in parquet format, but I'm confusing, because mapGroupWithState requires Update mode and parquet output requires Append mode. Is there any solution here ? Or can we use foreach sink someway for this case.

val query: Dataset[BidData] = bidStream
    .groupByKey(_.auction_id)
    .mapGroupsWithState(GroupStateTimeout.ProcessingTimeTimeout())(updateBidState)
    .flatMap(b => b)

query.writeStream
    .outputMode(OutputMode.Update())
    .format("parquet")
    .option("path", appConfig.s3Output)
    .option("checkpointLocation", appConfig.checkpoint)

Answer 1

Alternatively you can use flatMapGroupsWithState with append mode (mapGroupWithState is a special case of flatMapGroupsWithState).

But if you do so, your parquet file will contain all the historical bid states within it. If you need to get the last bid state you will have to write queries (I assume you will use SparkSQL or Hive) to return the last bid state.

Refer: https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KeyValueGroupedDataset-flatMapGroupsWithState.html