当Spark Streaming将在Dstream上执行输出操作时

Question

I was looking into the Spark Streaming programming Guide. I got one basic doubt like, when it will execute/compute the Dstream output operations. For example(I got it from one example):

val ssc = new StreamingContext(conf, Seconds(1))
val lines = ssc.socketTextStream("localhost", 7777) 
lines.foreachRDD { rdd =>
  rdd.foreachPartition { partitionOfRecords =>
    val connection = createNewConnection()
    partitionOfRecords.foreach(record => connection.send(record))
    connection.close()
  }
}
// Start the computation
ssc.start()
// Wait for the computation to terminate
ssc.awaitTermination()

Will it do the operation at each batch-iterval here 1 second. Or it wait till termination.

Answer 1

Will it do the operation at each batch-iterval here 1 second. Or it wait till termination.

It will read a batch at every 1 second interval and run the entire graph each time. In Spark terminology it's called executing a job at each interval.

The streaming job will only terminate once you single it to stop.