Spark结构化流存储绑定

Question

I am processing a stream of 100 Mb/s average load. I have six executors with each having 12 Gb of memory allocated. However, due to data load, I am getting Out of Memory errors (Error 52) in the spark executors in few minutes. It seems even though Spark dataframe is conceptually unbounded it is bounded by total executor memory?

My idea here was to save dataframe/stream as an in parquet in about every five minutes. However, it seems spark won't have a direct mechanism to purge the dataframe after that?

val out = df.
  writeStream.
  format("parquet").
  option("path", "/applications/data/parquet/customer").
  option("checkpointLocation", "/checkpoints/customer/checkpoint").
  trigger(Trigger.ProcessingTime(300.seconds)).
  outputMode(OutputMode.Append).
  start

Answer 1

It seems that there is no direct way to do this. As this conflicts with the general Spark model that operations be rerunnable in case of failure .

However I would share the same sentiment of the comment at 08/Feb/18 13:21 on this issue .