[英]Kafka Streams - ACLs for Internal Topics
I'm trying to setup a secure Kafka cluster and having a bit of difficulty with ACLs. 我正在尝试设置安全的Kafka集群,并且在使用ACL时遇到一些困难。
The Confluent security guide for Kafka Streams ( https://docs.confluent.io/current/streams/developer-guide/security.html ) simply states that the Cluster Create ACL has to be given to the principal... but it doesn't say anything about how to actually handle the internal topics. 关于Kafka Streams的Confluent安全指南( https://docs.confluent.io/current/streams/developer-guide/security.html )仅声明必须向主体提供集群创建ACL ...但是它没有关于如何实际处理内部主题什么也没说。
Through research and experimentation, I've determined (for Kafka version 1.0.0): 通过研究和实验,我确定(对于Kafka 1.0.0版):
Are the exact names of the internal topics predictable and consistent? 内部主题的确切名称是否可预测且一致? In other words, if I run my application on a dev server, will the exact same topics be created on the production server when run?
换句话说,如果我在开发服务器上运行应用程序,运行时是否会在生产服务器上创建完全相同的主题? If so, then I can just add ACLs derived from dev before deploying.
如果是这样,那么我可以在部署之前添加从dev派生的ACL。 If not, how should the ACLs be added?
如果没有,应该如何添加ACL?
Are the exact names of the internal topics predictable and consistent?
内部主题的确切名称是否可预测且一致? In other words, if I run my application on a dev server, will the exact same topics be created on the production server when run?
换句话说,如果我在开发服务器上运行应用程序,运行时是否会在生产服务器上创建完全相同的主题?
Yes, you'll get the same exact topics names from run to run. 是的,每次运行都会得到相同的主题名称。 The DSL generates processor names with a function that looks like this:
DSL生成具有以下功能的处理器名称:
public String newProcessorName(final String prefix) {
return prefix + String.format("%010d", index.getAndIncrement());
}
(where index
is just an incrementing integer). (其中
index
只是一个递增的整数)。 Those processor names are then used to create repartition topics with a function that looks like this (the parameter name
is a processor name generated as above): 然后使用这些处理器名称来创建具有以下功能的重新分区主题 (参数
name
是如上生成的处理器名称):
static <K1, V1> String createReparitionedSource(final InternalStreamsBuilder builder,
final Serde<K1> keySerde,
final Serde<V1> valSerde,
final String topicNamePrefix,
final String name) {
Serializer<K1> keySerializer = keySerde != null ? keySerde.serializer() : null;
Serializer<V1> valSerializer = valSerde != null ? valSerde.serializer() : null;
Deserializer<K1> keyDeserializer = keySerde != null ? keySerde.deserializer() : null;
Deserializer<V1> valDeserializer = valSerde != null ? valSerde.deserializer() : null;
String baseName = topicNamePrefix != null ? topicNamePrefix : name;
String repartitionTopic = baseName + REPARTITION_TOPIC_SUFFIX;
String sinkName = builder.newProcessorName(SINK_NAME);
String filterName = builder.newProcessorName(FILTER_NAME);
String sourceName = builder.newProcessorName(SOURCE_NAME);
builder.internalTopologyBuilder.addInternalTopic(repartitionTopic);
builder.internalTopologyBuilder.addProcessor(filterName, new KStreamFilter<>(new Predicate<K1, V1>() {
@Override
public boolean test(final K1 key, final V1 value) {
return key != null;
}
}, false), name);
builder.internalTopologyBuilder.addSink(sinkName, repartitionTopic, keySerializer, valSerializer,
null, filterName);
builder.internalTopologyBuilder.addSource(null, sourceName, new FailOnInvalidTimestamp(),
keyDeserializer, valDeserializer, repartitionTopic);
return sourceName;
}
If you don't change your topology—like, if don't change the order of how it's built, etc—you'll get the same results no matter where the topology is constructed (presuming you're using the same version of Kafka Streams). 如果您不更改拓扑(例如,不更改其构建顺序等),则无论在何处构建拓扑,都将获得相同的结果(假设您使用的是相同版本的Kafka流)。
If so, then I can just add ACLs derived from dev before deploying.
如果是这样,那么我可以在部署之前添加从dev派生的ACL。 If not, how should the ACLs be added?
如果没有,应该如何添加ACL?
I have not used ACLs, but I imagine that since these are just regular topics, then yeah, you can apply ACLs to them. 我没有使用过ACL,但是我想既然这些只是常规主题,那么可以将ACL应用于它们。 The security guide does mention:
安全指南确实提到:
When applications are run against a secured Kafka cluster, the principal running the application must have the ACL --cluster --operation Create set so that the application has the permissions to create internal topics.
当针对受保护的Kafka群集运行应用程序时,运行该应用程序的主体必须具有ACL --cluster --operation Create设置,以便该应用程序具有创建内部主题的权限。
I've been wondering about this myself, though, so if I am wrong I am guessing someone from Confluent will correct me. 我一直在想这个问题,所以,如果我错了,我想Confluent的某个人会纠正我。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.