How to increase Flink taskmanager.numberOfTaskSlots to run it without Flink server(in IDE or fat jar)

Question

I have one questions about running Flink streaming job in IDE or as fat jar without deploying it to Flink server.

The problem is I cannot run it in IDE when I have more than 1 taskslot in my job.

public class StreamingJob {

public static void main(String[] args) throws Exception {
    // set up the streaming execution environment
    final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    Properties kafkaProperties = new Properties();
    kafkaProperties.setProperty("bootstrap.servers", "localhost:9092");
    kafkaProperties.setProperty("group.id", "test");
    env.setParallelism(1);

    DataStream<String> kafkaSource = env
        .addSource(new FlinkKafkaConsumer010<>("flink-source", new SimpleStringSchema(), kafkaProperties))
        .name("Kafka-Source")
        .slotSharingGroup("Kafka-Source");

    kafkaSource.print().slotSharingGroup("Print");

    env.execute("Flink Streaming Java API Skeleton");
}

}

I know that job need 2 slot for this job and I can have two taskmanagers in Flink cluster, but how can I run it locally in IDE.

Currently I have to specify the same slotSharingGroup name for all operator locally to have one slot. But it's not flexible.

How do you handle it?

Answer 1

This is a known bug which you are describing. You can find the corresponding JIRA issue here .

The way to circumvent this problem is to manually set the number of task slots with which the TaskExecutor is started. You can do this via the TaskManagerOptions.NUM_TASK_SLOTS configuration option:

final int parallelism = ...;
final Configuration configuration = new Configuration();
configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 2);

final StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(parallelism, configuration);