在Kafka Streams应用程序中，有没有办法使用输出主题的通配符列表定义拓扑？

Question

I have multi-schema Kafka Streams application that enriches a record via a join to a KTable, and then passes the enriched record along.

The input topic naming format is currently well defined but I'm changing this to a wildcard. I want to determine the input topic of each record, derive the output topic via regex replacement, and send it on.

Eg While listening to event.raw.* a record comes in on event.raw.foo and I wish to pass it out on event.foo .

I realise I can get the input topics via the Processor API:

public class EnrichmentProcessor extends AbstractProcessor<String, GenericRecord> {

    @Override
    public void process(String key, GenericRecord value) {
        //Do Join...

        //Determine output topic and forward
        String outputTopic = context().topic().replaceFirst(".raw.", ".");
        context().forward(key, value, To.child(outputTopic));
        context().commit();
    }
}

But this doesn't help me when I'm trying to define my Topology because I have no way of knowing up front what my output topic is going to be.

  InternalTopologyBuilder topologyBuilder = new InternalTopologyBuilder();
        topologyBuilder.addSource("SOURCE", stringDeserializer, genericRecordDeserializer, "event.raw.*")
        .addProcessor("ENRICHER", EnrichmentProcessor::new, "SOURCE")
        .addSink("OUTPUT", outputTopic, stringSerializer, genericRecordSerializer, "ENRICHER"); // How can I register all possible output topics here?

Has anyone solved a situation like this before?

I know that if I had a list of possible output-topic names up front I could have multiple sinks defined on the topology but I'm not going to.

Is there a way I can define the topology to have dynamically allocated output topic names when I dont't have a hard coded list of possible output topic names up front?

Answer 1

This should be possible: You can use Topology#addSink(..., new TopicNameExtractor(){...}, ...) to dynamically set an output topic name. TopicNameExtractor has access to the RecordContext that allows you to get the input topic name via context.topic() . Hence, you should be able to compute the output topic name, base on the input topic name.