Spark Streaming 应用程序使用所有工作线程

Question

I have installed Apache spark on one machine with 3 workers and 2 cores for each worker.

Now I have two spark streaming applications. The first is written in Java and the other is written in python.

I need to run the two applications.

When I submit the first application, it uses all the workers. When I submit the second application it remains in the WAITING state as there is no available resources to run (as far as I understand).

Is there a way to allocate only two workers for the first application in order to leave space for the other application?

Answer 1

You should set the following configuration property in the spark configuration used to create the sparkContext .

spark.cores.max=<value>

to assign the right number of core for each spark job running. Keep in mind that if you are using the consumer-based streaming model, you need to reserve 1 core for each consumer you declare + additional cores for the spark processing.