can we start a new thread in foreachRDD in spark streaming?

Question

I want to start a child-thread in foreachRDD.

My situation is:

the job is reading from a hdfs dir continuously, and every 100 batches, I want to launch a model training task (I will make a snapshot of the rdds at that time and start the training task. the training task takes a very long time(2 hours), and I don't want the training task influence reading new batch of data.

Is starting a new child thread a good solution? Could the child thread use SparkContext in the main thread and use the rdd in main thread?

Answer 1

You don't need to start a new thread in RDD operations. To start a new job in every a hundred batch, you can add a BatchListner to count number of batches, and start a new job when number equals to 100. BatchListener Example