[英]Kafka Streams Internal Topics Redirection
Currently working with Kafka Streams to aggregate events in a client's system.目前正在使用 Kafka Streams 来聚合客户端系统中的事件。 When running our prototype with fake events, everything works perfectly.
当使用假事件运行我们的原型时,一切正常。 However, when using actual data, we noticed that in the process of aggregation, Streams automatically creates internal topics.
但是,在使用实际数据时,我们注意到在聚合过程中,Streams 会自动创建内部主题。 While in theory this is fine, our client has necessary, super tight security and is unwilling to grant my development team topic creating privileges.
虽然理论上这很好,但我们的客户有必要的、超级严格的安全性,并且不愿意授予我的开发团队主题创建权限。 This means we cannot run our Streams program as-is.
这意味着我们不能按原样运行我们的 Streams 程序。
We can, though, have topics created for us and use those instead of Streams creating its own Kafka topics.但是,我们可以为我们创建主题并使用这些主题而不是 Streams 创建自己的 Kafka 主题。 Is it possible/how would one start to go about redirecting Streams internal topic creation to leverage existing topics?
是否有可能/如何开始 go 关于重定向 Streams 内部主题创建以利用现有主题?
Note: We can name the internal topics whatever we want.注意:我们可以随意命名内部主题。 It just has to be created by the team that has those privileges.
它只需要由拥有这些特权的团队创建。
In Kafka Streams, there are now overloaded methods for both KStream and KTable that accept a new parameter Named.在 Kafka Streams 中,现在 KStream 和 KTable 都有重载方法,它们接受一个新的参数 Named。 By using the Named class DSL, users can provide meaningful names to the processors in their topology.
通过使用命名的 class DSL,用户可以为其拓扑中的处理器提供有意义的名称。
KStream<String,String> stream =
builder.stream("input", Consumed.as("Customer_transactions_input_topic"));
stream.filter((k,v) -> !v.equals("invalid_txn"), Named.as("filter_out_invalid_txns"))
.mapValues((v) -> v.substring(0,5), Named.as("Map_values_to_first_6_characters"))
.to("output", Produced.as("Mapped_transactions_output_topic"));
Topologies:
Sub-topology: 0
Source: Customer_transactions_input_topic (topics: [input])
--> filter_out_invalid_txns
Processor: filter_out_invalid_txns (stores: [])
--> Map_values_to_first_6_characters
<-- Customer_transactions_input_topic
Processor: Map_values_to_first_6_characters (stores: [])
--> Mapped_transactions_output_topic
<-- filter_out_invalid_txns
Sink: Mapped_transactions_output_topic (topic: output)
<-- Map_values_to_first_6_characters
Now, take a look at your topology with all the processors named:现在,看看你的拓扑结构,所有处理器命名为:
Now you can look at the topology description and easily understand what role each processor plays in the topology.现在您可以查看拓扑描述并轻松了解每个处理器在拓扑中所扮演的角色。 But there's another reason for naming your processor nodes when you have stateful operators that remain between restarts of your Kafka Streams applications, state stores, changelog topics, and repartition topics, which has to do with potential name shifting of the processor nodes that use generated names.
但是,当您在 Kafka Streams 应用程序、state 存储、更改日志主题和重新分区主题之间保留有状态操作符时,命名您的处理器节点还有另一个原因,这与使用生成名称的处理器节点的潜在名称转换有关.
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.