Spark streaming uses lesser number of executors

Question

I am using spark streaming process some events. It is deployed in standalone mode with 1 master and 3 workers. I have set number of cores per executor to 4 and total num of executors to 24. This means totally 6 executors will be spawned. I have set spread-out to true. So each worker machine get 2 executors. My batch interval is 1 second. Also I have repartitioned the batch to 21. The rest 3 are for receivers. While running what I observe from event timeline is that only 3 of the executors are being used. The other 3 are not being used. As far as I know, there is no parameter in spark standalone mode to specify the number of executors. How do I make spark to use all the available executors?

Answer 1

Probably your streaming has not so many partitions to fill all executors on every 1-second minibatch. Try repartition(24) as first streaming transformation to use full spark cluster power.