具有不同主题 ApplicationId 的多个 Kafka 流

Question

Given a single component which can have multiple instances, and the following structure:

1. Stream1[Topic1, destination1]
2. Stream2[Topic2, destination2]

where destination is a Queue and all the links will be 1:1.

Do we need to set the same applicationId for each KafkaStream? It is known that applicationId will generate client.id and group.id which are important for how the partitions are assigned. Couldn't find anything in the official documentation.

Answer 1

You can run two applications in the same JVM process with separate threads for starting both topologies, or you can simply run two independent JVM processes. Both cases, use different ids.

Or you can run one process (one id), subscribe to both topics, but use branch operator to separate streams by topic names.

• https://www.confluent.io/blog/putting-events-in-their-place-with-dynamic-routing/
• https://docs.confluent.io/platform/current/streams/developer-guide/dsl-api.html#stateless-transformations

Answer 2

If your program is different (ie, different Topology ), you need to use different application.id config. Using the same application.id config requires that all instances execute the exact same Topology .

Of course, you could also build a single Topology that processed both topics at once:

StreamsBuilder builder = ...

builder.stream("topic1")...to("destination1");
builder.stream("topic2")...to("destination2");