[英]Apache Kafka create topic from code
As we know Topic
creation in Kafka should be handled on the the server initialization part. 我们知道Kafka中的
Topic
创建应该在服务器初始化部分进行处理。 There we use the default script ./kafka-topics --zookeeper ...
, but what if we need to create a topic dynamically? 我们使用默认脚本
./kafka-topics --zookeeper ...
,但是如果我们需要动态创建主题呢?
Fortunately, Kafka 0.10.1.0
brought us this ability. 幸运的是,
Kafka 0.10.1.0
为我们带来了这种能力。 I saw these fascinating feature on the Confluence Jira board but couldn't find any documentation related to the topic, irony, isn't it? 我在Confluence Jira板上看到了这些引人入胜的功能,但找不到与该主题相关的任何文档,具有讽刺意味,不是吗?
So, I went to the source code and found the way of creating topics on the fly. 所以,我去了源代码,找到了动态创建主题的方法。 Hopefully it will be helpful for some of you.
希望它对你们中的一些人有所帮助。 Of course, if you have a better solution, please, do not hesitate to share it with us.
当然,如果您有更好的解决方案,请不要犹豫与我们分享。
Ok, let's start. 好的,我们开始吧。
/** The method propagate topics **/
public List<String> propagateTopics(int partitions, short replication, int timeout) throws IOException {
CreateTopicsRequest.TopicDetails topicDetails = new CreateTopicsRequest.TopicDetails(partitions, replication);
Map<String, CreateTopicsRequest.TopicDetails> topicConfig = mTopics.stream()
.collect(Collectors.toMap(k -> k, v -> topicDetails)); // 1
CreateTopicsRequest request = new CreateTopicsRequest(topicConfig, timeout); // 2
try {
CreateTopicsResponse response = createTopic(request, BOOTSTRAP_SERVERS_CONFIG); // 3
return response.errors().entrySet().stream()
.filter(error -> error.getValue() == Errors.NONE)
.map(Map.Entry::getKey)
.collect(Collectors.toList()); // 4
} catch (IOException e) {
log.error(e);
}
return null;
}
1
we need an instance ofTopicDetails
, for simplicity, I'll share the same configs among all topics.1
我们需要一个TopicDetails
实例,为简单起见,我将在所有主题中共享相同的配置。 Assume, thatmTopics
is your list of Strings of all topics you want to create.假设
mTopics
是您要创建的所有主题的字符串列表。
2
Basically we want to send a request to our Kafka cluster, now we have the special class for that, - that acceptsCreateTopicsRequest
and timeout2
基本上我们想要向我们的Kafka集群发送请求,现在我们有了特殊的类, - 接受CreateTopicsRequest
和timeout
3
Than we need to send the request and get theCreateTopicsResponse
3
我们需要发送请求并获取CreateTopicsResponse
private static final short apiKey = ApiKeys.CREATE_TOPICS.id;
private static final short version = 0;
private static final short correlationId = -1;
private static CreateTopicsResponse createTopic(CreateTopicsRequest request, String client) throws IllegalArgumentException, IOException {
String[] comp = client.split(":");
if (comp.length != 2) {
throw new IllegalArgumentException("Wrong client directive");
}
String address = comp[0];
int port = Integer.parseInt(comp[1]);
RequestHeader header = new RequestHeader(apiKey, version, client, correlationId);
ByteBuffer buffer = ByteBuffer.allocate(header.sizeOf() + request.sizeOf());
header.writeTo(buffer);
request.writeTo(buffer);
byte byteBuf[] = buffer.array();
byte[] resp = requestAndReceive(byteBuf, address, port);
ByteBuffer respBuffer = ByteBuffer.wrap(resp);
ResponseHeader.parse(respBuffer);
return CreateTopicsResponse.parse(respBuffer);
}
private static byte[] requestAndReceive(byte[] buffer, String address, int port) throws IOException {
try(Socket socket = new Socket(address, port);
DataOutputStream dos = new DataOutputStream(socket.getOutputStream());
DataInputStream dis = new DataInputStream(socket.getInputStream())
) {
dos.writeInt(buffer.length);
dos.write(buffer);
dos.flush();
byte resp[] = new byte[dis.readInt()];
dis.readFully(resp);
return resp;
} catch (IOException e) {
log.error(e);
}
return new byte[0];
}
Here is no magic at all, just sending the request, and than parsing the byte stream to the response. 这根本不是魔术,只是发送请求,而不是将字节流解析为响应。
4
CreateTopicsResponse
has propertyerrors
, which is just aMap<String, Errors>
wherekey
is the topic name you requested.4
CreateTopicsResponse
有属性errors
,它只是Map<String, Errors>
,其中key
是您请求的主题名称。 The tricky thing, it contains all topics you requested, but those with no errors has valueErrors.None
, that's why I'm filtering the response and return only successfully created topics.棘手的是,它包含您请求的所有主题,但没有错误的主题具有值
Errors.None
,这就是我过滤响应并仅返回成功创建的主题的原因。
Extending Andrei Nechaev answers 延伸Andrei Nechaev的答案
With 10.2.0, the way to get an instance of CreateTopicsRequest has changed a bit. 在10.2.0中,获取CreateTopicsRequest实例的方式有所改变。 We need to use the Builder inner class to build a CreateTopicsRequest instance.
我们需要使用Builder内部类来构建CreateTopicsRequest实例。 Here is a code sample.
这是一个代码示例。
CreateTopicsRequest.Builder builder = new CreateTopicsRequest.Builder(topicConfig, timeout, false);
CreateTopicsRequest request = builder.build();
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.