spark-streaming 是否并行运行多个 foreach

Question

In this case

val dStream : Stream[_] = 
dStream.foreachRDD(a => ... )
dStream.foreachRDD(b => ... )

Do the foreach methods :

1. run in parallel?
2. run in sequence but without specific order?
3. The foreachRDD( a => ) before the foreachRDD( b => )?

I want to know that because I want to commit kafka offset after a database insert. ( And the db connector give just a "foreach" insert )

val dStream : Stream[_] = ...().cache()
dStream.toDb // consume the stream
dStream.foreachRDD(b => //commit offset ) //consume the stream but after the db insert

In the spark UI it's look like there is an order but I'm not sure it's reliable.

Edit : if foreachRDD( a => ) fail , do the foreachRDD( b => ) is still executed?

Answer 1

DStream.foreach is deprecated since Spark 0.9.0. You want the equivalent DStream.foreachRDD to begin with.

Stages in the Spark DAG are executed sequentially, as one transformation's output is usually also the input for the next transformation in the graph, but this isn't the case in your example.

What happens is that internally the RDD is divided into partitions. Each partition is ran on a different worker which is available to the cluster manager. In your example, DStream.foreach(a => ...) will execute before DStream.foreach(b => ...) , but the execution within foreach will run in parallel as regards to the internal RDD being iterated.

I want to know that because I want to commit kafka offset after a database insert.

The DStream.foreachRDD is an output transformation, meaning it will cause the Spark to materialize the graph and begin execution. You can safely assume that the insertion to the database will end prior to executing your second foreach , but keep in mind that your first foreach will be updating your database in parallel foreach partition in the RDD .

Answer 2

Multiple DStream.foreachRDD are not guaranteed to execute sequentially atleast till spark-streaming-2.4.0. Look at this code in JobScheduler class:

class JobScheduler(val ssc: StreamingContext) extends Logging {

  // Use of ConcurrentHashMap.keySet later causes an odd runtime problem due to Java 7/8 diff
  // https://gist.github.com/AlainODea/1375759b8720a3f9f094
  private val jobSets: java.util.Map[Time, JobSet] = new ConcurrentHashMap[Time, JobSet]
  private val numConcurrentJobs = ssc.conf.getInt("spark.streaming.concurrentJobs", 1)
  private val jobExecutor =
    ThreadUtils.newDaemonFixedThreadPool(numConcurrentJobs, "streaming-job-executor")

The JobExecutor is a thread pool and if "spark.streaming.concurrentJobs" is set to a number more than 1, there can be parallel execution if enough spark-executors are available. So make sure your settings are correct to elicit a behavior you need.