Dynamic Streams Topology in Kafka

Question

While creating Kafka Streams using Kafka Streams DSL https://kafka.apache.org/0110/documentation/streams/developer-guide

we have encountered a scenario where we need to update the Kafka Streams with new topology definition.

For example: When we started, we have a topology defined to read from one topic (Source) and a destination topic (Sink). However, on a configuration change we now need to read from 2 different topics (2 sources if you will) and write to a single destination topic.

From what we have built right now, the topology definition is hard coded, something like defined in processor topology .

Questions:

1. Is it possible to define topology in a declarative way (say in a Json or something else), which doesn't require a codification of the topology?

2. Is it possible to reload an existing Kafka Stream to use a new definition of the Kafka Streams Topology?

3. For #2 mentioned above, does Kafka Streams DSL provide a way to "reload" new topology definitions by way of an external trigger or system call?

We are using JDK 1.8 and Kafka DSL 2.2.0

Thanks, Ayusman

Answer 1

Is it possible to define topology in a declarative way (say in a Json or something else), which doesn't require a codification of the topology?

The KStreams DSL is declarative, but I assume you mean something other than the DSL?

If so, the answer is No. You may want to look at KSQL, however.

Is it possible to reload an existing Kafka Stream to use a new definition of the Kafka Streams Topology?

You mean if an existing Kafka Streams application can reload a new definition of a processing topology? If so, the answer is No. In such cases, you'd deploy a new version of your application.

Depending on how the old/new topologies are defined, a simple rolling upgrade of your application may suffice (roughly: if the topology change was minimal), but probably you will need to deploy the new application separately and then, once the new one is vetted, decommission your old application.

Note: KStreams is a Java library and, by design, does not include functionality to operate/manage the Java applications that use the KStreams library.

For #2 mentioned above, does Kafka Streams DSL provide a way to "reload" new topology definitions by way of an external trigger or system call?

No.