将数据从 Spark Structured Streaming 加载到 ArrayList

Question

I need to send data from Kafka to Kinesis Firehose. I am processing Kafka data using Spark Structured Streaming. I am not sure how to process the dataset of streaming query into an ArrayList variable - say, recordList - of eg 100 records (could be any other value) and then call Firehose API's putRecordBatch(recordList) to put the records into Firehose.

Answer 1

I think you want to check out Foreach and ForeachBatch depending on your Spark Version. ForeachBatch comes in V2.4.0 and foreach is available < V2.4.0. If there is no streaming sink implementation available for Kinesis Firehouse then you should make your own implementation of the ForeachWriter . Databricks has some nice examples of using foreach to create the custom writers.

I haven't ever used Kinesis but here is an example of what your custom sink might look like.

case class MyConfigInfo(info1: String, info2: String)

class  KinesisSink(configInfo: MyConfigInfo) extends ForeachWriter[(String, String)] {
  val kinesisProducer = _

  def open(partitionId: Long,version: Long): Boolean = {
    kinesisProducer = //set up the kinesis producer using MyConfigInfo
      true
  }

  def process(value: (String, String)): Unit = {
    //ask kinesisProducer to send data
  }

  def close(errorOrNull: Throwable): Unit = {
    //close the kinesis producer
  }
}

if you're using that AWS kinesisfirehose API you might do something like this

case class MyConfigInfo(info1: String, info2: String)

class  KinesisSink(configInfo: MyConfigInfo) extends ForeachWriter[(String, String)] {
  val firehoseClient = _
  val req = putRecordBatchRequest = new PutRecordBatchRequest()
  val records = 0
  val recordLimit = //maybe you need to set this? 

  def open(partitionId: Long,version: Long): Boolean = {
    firehoseClient = //set up the firehose client using MyConfigInfo
      true
  }

  def process(value: (String, String)): Unit = {
    //ask fireHose client to send data or batch the request
    val record: Record = //create Record out of value
    req.setRecords(record)
    records = records + 1
    if(records >= recordLimit) {
      firehoseClient.putRecordBatch(req)
      records = 0
    }
  }

  def close(errorOrNull: Throwable): Unit = {
    //close the firehose client
    //or instead you could put the batch request to the firehose client here but i'm not sure if that's good practice
  }
}

Then you'd use it as such

val writer = new KinesisSink(configuration)
val query =
  streamingSelectDF
    .writeStream
    .foreach(writer)
    .outputMode("update")
    .trigger(ProcessingTime("25 seconds"))
    .start()