Kafka Streams produce messages conditionally after transformation

Question

We have a use-case, where we have to read some messages into the KStreams, then transform the message and produce it to another topic conditionally.

In our use-case, for transformation of object, we make an downstream API call. If the API call is succesfull, then produce to newTopic1, else produce to newTopic2. How can the same be achieved ??

As of now, we are using the below style of producing the enriched (ie transformed objects) to the new Kafka topic using to method provided by Streams API.

KStream<String, Customer> transformedStream = sourceKafkaStream
                .mapValues(cust -> {
                    try {
                        logger.info("Hitting to the downstream to fetch additional information, will take few seconds.");
                        Thread.sleep(7000);
                        return RecordEnrichmentAndTransformationBuilder.enrichTheCustomer(cust);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                    return cust;
                });
                .to('newTopic1', Produced.with(AppSerdes.String(), AppSerdes.serdeForEnrichedCustomer()));

Appreciate the response on this.

Answer 1

Using DSL api you use KStream::filter or KStream:to(TopicNameExtractor<K, V> topicExtractor, Produced<K, V> produced) .

Sample code will looks like that if both format are the same:

KStream<String, Customer> transformedStream = sourceKafkaStream
                .mapValues(cust -> {
                    try {
                        logger.info("Hitting to the downstream to fetch additional information, will take few seconds.");
                        Thread.sleep(7000);
                        return RecordEnrichmentAndTransformationBuilder.enrichTheCustomer(cust);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                    return cust;
                });
                .to((key, value, recordContext) -> topicNameCalculation(key, value), Produced.with(AppSerdes.String(), AppSerdes.serdeForEnrichedCustomer()));

topicNameCalculation(...) will based on the key and the value chose the proper topic.

One Notice Generally it is not good approach to make external call in Kafka Streams.