本地模式下Apache Flink任务管理器的数量

Question

I am working on an Apache Flink (1.5.0) based streaming application. As part of this I have launched Flink in local mode on my Windows machine.

In order to run my job with the degree of parallelism of 8, I need 8 Task managers providing one task slot each.

I added a task manager with following command:

/cygdrive/b/Binaries Flink/flink-1.5.0/bin/taskmanager.sh' start

The first few times, a task manager was added successfully with following message:

[INFO] 3 instance(s) of taskexecutor are already running on ... .
Starting taskexecutor daemon on host ... .

After 5 task managers were available I got the same message

[INFO] 5 instance(s) of taskexecutor are already running on ... .
Starting taskexecutor daemon on host ... .

The problem is that a sixth task manager is never created. When I stop one task manager it goes down to 4, I can add one additional task manager but never more than 5 task managers.

Is there any limitation to the amount of task managers? Did anyone experience a similar behaviour?

Thank you very much

Answer 1

There is no limit of how many TaskManager you can start locally. The only limit is the available resources you have on your local machine.

If you are using the standalone mode in Flink 1.5.0 , then you can also set the number of slots per TaskManager to 7 by adding the following line to the flink-conf.yaml :

taskmanager.numberOfTaskSlots: 7