[英]How to configure kafka topic retention policy during creation with Spring?
I need to configure retention policy of a particular topic during creation.我需要在创建期间配置特定主题的保留策略。 I tried to look for solution i could only find command level alter command as below
我试图寻找解决方案,我只能找到如下命令级别的更改命令
./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.ms=1680000
./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config reserved.ms=1680000
Can someone let me know a way to configure it during creation, something like xml or properties file configuration in spring-mvc.有人可以让我知道在创建过程中配置它的方法,例如 spring-mvc 中的 xml 或属性文件配置。
Spring Kafka lets you create new topics by declaring @Bean
s in your application context. Spring Kafka 允许您通过在应用程序上下文中声明
@Bean
创建新主题。 This will require a bean of type KafkaAdmin
in the application context, which will be created automatically if using Spring Boot.这将需要应用程序上下文中的
KafkaAdmin
类型的 bean,如果使用 Spring Boot,它将自动创建。 You could define your topic as follows:您可以按如下方式定义您的主题:
@Bean
public NewTopic myTopic() {
return TopicBuilder.name("my-topic")
.partitions(4)
.replicas(3)
.config(TopicConfig.RETENTION_MS_CONFIG, "1680000")
.build();
}
If you are not using Spring Boot, you'll additionally have to define the KafkaAdmin
bean:如果您不使用 Spring Boot,则还必须定义
KafkaAdmin
bean:
@Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
return new KafkaAdmin(configs);
}
If you want to edit the configuration of an existing topic, you'll have to use the AdminClient
, here's the snippet to change the retention.ms
at a topic level:如果要编辑现有主题的配置,则必须使用
AdminClient
,这是在主题级别更改retention.ms
AdminClient
的代码段:
Map<String, Object> config = new HashMap<>();
config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
AdminClient client = AdminClient.create(config);
ConfigResource resource = new ConfigResource(ConfigResource.Type.TOPIC, "new-topic");
// Update the retention.ms value
ConfigEntry retentionEntry = new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, "1680000");
Map<ConfigResource, Config> updateConfig = new HashMap<>();
updateConfig.put(resource, new Config(Collections.singleton(retentionEntry)));
AlterConfigOp op = new AlterConfigOp(retentionEntry, AlterConfigOp.OpType.SET);
Map<ConfigResource, Collection<AlterConfigOp>> configs = new HashMap<>(1);
configs.put(resource, Arrays.asList(op));
AlterConfigsResult alterConfigsResult = client.incrementalAlterConfigs(configs);
alterConfigsResult.all();
The configuration can be set up automatically using this @PostConstruct
method that takes in NewTopic
beans.可以使用这个
@PostConstruct
方法自动设置配置,该方法接受NewTopic
bean。
@Autowired
private Set<NewTopic> topics;
@PostConstruct
public void reconfigureTopics() throws ExecutionException, InterruptedException {
try (final AdminClient adminClient = AdminClient.create(Map.of(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBootstrapServers))) {
adminClient.incrementalAlterConfigs(topics.stream()
.filter(topic -> topic.configs() != null)
.collect(Collectors.toMap(
topic -> new ConfigResource(ConfigResource.Type.TOPIC, topic.name()),
topic -> topic.configs().entrySet()
.stream()
.map(e -> new ConfigEntry(e.getKey(), e.getValue()))
.peek(ce -> log.debug("configuring {} {} = {}", topic.name(), ce.name(), ce.value()))
.map(ce -> new AlterConfigOp(ce, AlterConfigOp.OpType.SET))
.collect(Collectors.toList())
)))
.all()
.get();
}
}
I guess you could use admin client ( https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html ) for this.我想您可以为此使用管理客户端( https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html )。 You can create Admin client instance in your application and use create or alter topic command for manipulating topic configurations, including retention.
您可以在应用程序中创建管理客户端实例,并使用 create 或 alter topic 命令来操作主题配置,包括保留。
To create a topic using AdminClient
programmatically with the specified retention time, do the following:要使用
AdminClient
编程方式创建具有指定保留时间的主题,请执行以下操作:
NewTopic topic = new NewTopic(topicName, numPartitions, replicationFactor);
topic.configs(Map.of(TopicConfig.RETENTION_MS_CONFIG, retentionMs.toString()));
adminClient.createTopics(List.of(topic));
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.