简体繁体 English

Kafka中的动态流拓扑

[英]Dynamic Streams Topology in Kafka

原文 2019-06-18 06:26:25 0 1 java/ apache-kafka/ apache-kafka-streams

While creating Kafka Streams using Kafka Streams DSL https://kafka.apache.org/0110/documentation/streams/developer-guide 使用Kafka Streams DSL创建Kafka Streams时https://kafka.apache.org/0110/documentation/streams/developer-guide

we have encountered a scenario where we need to update the Kafka Streams with new topology definition. 我们遇到了需要用新的拓扑定义更新Kafka流的情况。

For example: When we started, we have a topology defined to read from one topic (Source) and a destination topic (Sink). 例如：开始时，我们定义了一个拓扑以读取一个主题（源）和目标主题（接收器）。 However, on a configuration change we now need to read from 2 different topics (2 sources if you will) and write to a single destination topic. 但是，在配置更改时，我们现在需要阅读2个不同的主题（如果需要，请阅读2个来源）并写入单个目标主题。

From what we have built right now, the topology definition is hard coded, something like defined in processor topology . 从我们现在已经构建的内容来看，拓扑定义是硬编码的，类似于处理器拓扑中的定义。

Questions: 问题：

Is it possible to define topology in a declarative way (say in a Json or something else), which doesn't require a codification of the topology? 是否可以以声明性的方式（例如在Json或其他方式中）定义拓扑，而无需对拓扑进行编码？
Is it possible to reload an existing Kafka Stream to use a new definition of the Kafka Streams Topology? 是否可以重新加载现有的Kafka Stream以使用Kafka Streams拓扑的新定义？
For #2 mentioned above, does Kafka Streams DSL provide a way to "reload" new topology definitions by way of an external trigger or system call? 对于上述＃2，Kafka Streams DSL是否提供一种通过外部触发器或系统调用“重新加载”新拓扑定义的方法？

We are using JDK 1.8 and Kafka DSL 2.2.0 我们正在使用JDK 1.8和Kafka DSL 2.2.0

Thanks, Ayusman 谢谢，Ayusman

1 个解决方案

Is it possible to define topology in a declarative way (say in a Json or something else), which doesn't require a codification of the topology? 是否可以以声明性的方式（例如在Json或其他方式中）定义拓扑，而无需对拓扑进行编码？

The KStreams DSL is declarative, but I assume you mean something other than the DSL? KStreams DSL是声明性的，但我认为您的意思不是DSL？

If so, the answer is No. You may want to look at KSQL, however. 如果是这样，答案是否定的。但是，您可能要看一下KSQL。

Is it possible to reload an existing Kafka Stream to use a new definition of the Kafka Streams Topology? 是否可以重新加载现有的Kafka Stream以使用Kafka Streams拓扑的新定义？

You mean if an existing Kafka Streams application can reload a new definition of a processing topology? 您的意思是说，现有的Kafka Streams 应用程序是否可以重新加载处理拓扑的新定义？ If so, the answer is No. In such cases, you'd deploy a new version of your application. 如果是这样，答案是否定的。在这种情况下，您将部署新版本的应用程序。

Depending on how the old/new topologies are defined, a simple rolling upgrade of your application may suffice (roughly: if the topology change was minimal), but probably you will need to deploy the new application separately and then, once the new one is vetted, decommission your old application. 根据旧/新拓扑的定义方式，对应用程序进行简单的滚动升级就足够了（大约：如果拓扑更改很小），但是可能需要分别部署新应用程序，然后在新应用程序部署完毕后进行部署。经过审查，停用旧应用程序。

Note: KStreams is a Java library and, by design, does not include functionality to operate/manage the Java applications that use the KStreams library. 注意：KStreams是一个Java库，从设计上讲，它不包括操作/管理使用KStreams库的Java应用程序的功能。

For #2 mentioned above, does Kafka Streams DSL provide a way to "reload" new topology definitions by way of an external trigger or system call? 对于上述＃2，Kafka Streams DSL是否提供一种通过外部触发器或系统调用“重新加载”新拓扑定义的方法？

No. 没有。