Flink: all traffic goes to one Task Manager (in cluster with 1 Job Manager + 2 Task Managers)

Question

I have following set up:

Installation type k8s: 1.18
Flink version: 1.12
1 Job Manager
2 Task Manager

In flink-conf.yaml of task manager

  flink-conf.yaml: |
    state.backend: rocksdb
    blob.server.port: 6124
    jobmanager.rpc.port: 6123
    parallelism.default: 2
    queryable-state.proxy.ports: 6125
    taskmanager.numberOfTaskSlots: 2
    taskmanager.rpc.port: 6122
    jobmanager.memory.process.size: 2900m
    taskmanager.memory.process.size: 2900m
    jobmanager.web.address: 0.0.0.0
    rest.address: 0.0.0.0
    rest.bind-address: 0.0.0.0

In flink-conf.yaml of job manager

  flink-conf.yaml: |
    state.backend: rocksdb
    blob.server.port: 6124
    jobmanager.rpc.port: 6123
    parallelism.default: 2
    queryable-state.proxy.ports: 6125
    taskmanager.rpc.port: 6122
    jobmanager.memory.process.size: 2900m
    taskmanager.memory.process.size: 2900m
    jobmanager.web.address: 0.0.0.0
    rest.address: 0.0.0.0
    rest.bind-address: 0.0.0.0
    rest.port: 8081

With above configuration, only one task manager is active ie gets traffic and another task manager remains idle, even though number of events increases to extreme level.

Please suggest, if I am missing anything?

Answer 1

You have set the parallelism to 2, and given each task manager 2 slots. Thus a single task manager can provide the requested parallelism, and that's what will happen by default.

If you want the scheduler to behave differently, you could set

cluster.evenly-spread-out-slots: true

or you could reduce taskmanager.numberOfTaskSlots to 1.