[英]How to increase Flink taskmanager.numberOfTaskSlots to run it without Flink server(in IDE or fat jar)
I have one questions about running Flink streaming job in IDE or as fat jar without deploying it to Flink server. 我有一个关于在IDE中运行Flink流媒体作业的问题,或者没有将其部署到Flink服务器的胖jar。
The problem is I cannot run it in IDE when I have more than 1 taskslot in my job. 问题是当我在我的工作中有超过1个任务时,我无法在IDE中运行它。
public class StreamingJob {
public static void main(String[] args) throws Exception {
// set up the streaming execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties kafkaProperties = new Properties();
kafkaProperties.setProperty("bootstrap.servers", "localhost:9092");
kafkaProperties.setProperty("group.id", "test");
env.setParallelism(1);
DataStream<String> kafkaSource = env
.addSource(new FlinkKafkaConsumer010<>("flink-source", new SimpleStringSchema(), kafkaProperties))
.name("Kafka-Source")
.slotSharingGroup("Kafka-Source");
kafkaSource.print().slotSharingGroup("Print");
env.execute("Flink Streaming Java API Skeleton");
}
} }
I know that job need 2 slot for this job and I can have two taskmanagers in Flink cluster, but how can I run it locally in IDE. 我知道这个工作需要2个插槽,我可以在Flink集群中有两个任务管理器,但是如何在IDE中本地运行它。
Currently I have to specify the same slotSharingGroup name for all operator locally to have one slot. 目前,我必须在本地为所有运营商指定相同的slotSharingGroup名称,以便拥有一个插槽。 But it's not flexible.
但它并不灵活。
How do you handle it? 你怎么处理它?
This is a known bug which you are describing. 这是您正在描述的已知错误。 You can find the corresponding JIRA issue here .
您可以在此处找到相应的JIRA问题。
The way to circumvent this problem is to manually set the number of task slots with which the TaskExecutor
is started. 解决此问题的方法是手动设置启动
TaskExecutor
的任务槽数。 You can do this via the TaskManagerOptions.NUM_TASK_SLOTS
configuration option: 您可以通过
TaskManagerOptions.NUM_TASK_SLOTS
配置选项执行此操作:
final int parallelism = ...;
final Configuration configuration = new Configuration();
configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, 2);
final StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(parallelism, configuration);
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.