How to increase number of taskmanagers in flink 1.9.0 or latest versions

Question

I have 1452 independent tasks in a flink job. It reads from kafka. Then do some transformation using flatmap and then sinks in hdfs files. Kafka and and flatmap has parallelism of 20 each and i have 1450 independent sink each having parallelism of 1. As maximum parallelism is 20 here. Flink creates only 5 taskmanagers when i use 4 slots per taskmanagers. As number of total tasks in job is very high i need more taskmanagers to be created.

As of now i am giving more parallelism(100) to one of the sink task so that i get desired number of taskmanagers(which is not a proper way) but it leads to all sink tasks(except the one with 100 task slots) to be created on only one taskmanager and remaning taskmanager is used by other tasks.

So i need some way to instantiate desired number of taskmanagers in fink and some way to distribute sink tasks into all taskmanagers.

Answer 1

The simpliest way to isolate slots and force the YARN to provide additional task managers and slots will be to use Slot Sharing Groups for operations you want to isolate. By default all operations are put into the default slot sharing group and therefore all your tasks share the same slot.

To do it, just specify a different slot sharing group for different jobs after operators you want to isolate:

strean
   .op(...)
   .slotSharingGroup("job-N")