Kafka Streams KTable to Stream INVALID_TOPIC_EXCEPTION

Question

I have a Kafka Stream service that for apparently no reason starts to have a specific error even in local runnings.

My Stream service does few operations:

stream
  .map(doSomething)
  .filter(filterSomething)
  .groupBy(groupMyStuffs)
  .aggregate(Map.empty[String, Object])(aggregation)
  .mapValues((k, v) => parseAggResults(k, v))
  .toStream
  .flatMap((_, v) => v)
  .to(outputTopic)

Well, doing my tests I identify that my service was breaking after the mapValues when it call to the function toStream that will write my data to a new topic created by the Kafka Streams converting the KTable to Kafka Stream.

I checked the topics that KStreams create, and the topics are there:

myconsumergroup-KSTREAM-AGGREGATE-STATE-STORE-0000000005-changelog
myconsumergroup-KSTREAM-AGGREGATE-STATE-STORE-0000000005-repartition

Then I tried to go a little bit further in the problem. I started to debug the Kafka Clients classes to identify why this error is appearing and I found that at this line of code:

https://github.com/apache/kafka/blob/2.0/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L966

I was able to find that the response has three inputs as topicMetadata and I don't know what is this third one:

(type=TopicMetadata, error=NONE, topic=myconsumergroup-KSTREAM-AGGREGATE-STATE-STORE-0000000005-repartition, isInternal=false, partitionMetadata=[(type=PartitionMetadata, error=NONE, partition=0, leader=xxx.xxx.xxx.xxx:9092 (id: 0 rack: null), replicas=xxx.xxx.xxx.xxx:9092 (id: 0 rack: null), isr=xxx.xxx.xxx.xxx:9092 (id: 0 rack: null), offlineReplicas=)])
(type=TopicMetadata, error=NONE, topic=myconsumergroup-KSTREAM-AGGREGATE-STATE-STORE-0000000005-changelog, isInternal=false, partitionMetadata=[(type=PartitionMetadata, error=NONE, partition=0, leader=xxx.xxx.xxx.xxx:9092 (id: 0 rack: null), replicas=xxx.xxx.xxx.xxx:9092 (id: 0 rack: null), isr=xxx.xxx.xxx.xxx:9092 (id: 0 rack: null), offlineReplicas=)])
(type=TopicMetadata, error=INVALID_TOPIC_EXCEPTION, topic=, isInternal=false, partitionMetadata=[])

And just to make sure everything is cover here is the configuration that I have:

logger.info(s"Loading Kafka configurations")
logger.info(s"Kafka Connection with: ${getEnvVar("KAFKA_PROTOBUF_CONN")}")
logger.info(s"Consumer Name: ${getEnvVar("CONSUMER_STREAM_NAME")}")
settings.put(StreamsConfig.APPLICATION_ID_CONFIG, getEnvVar("CONSUMER_STREAM_NAME"))
settings.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, getEnvVar("KAFKA_PROTOBUF_CONN"))
settings.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, getEnvVar("AUTO_OFFSET_RESET_CONFIG"))
settings.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass.getName)
settings.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.ByteArray().getClass.getName)
settings.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10.seconds)
settings.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, "0")
//Added to avoid messages created by old producers
settings.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, "org.apache.kafka.streams.processor.WallclockTimestampExtractor")

settings.put(StreamsConfig.BUFFERED_RECORDS_PER_PARTITION_CONFIG, "1000")
settings.put(StreamsConfig.producerPrefix(ProducerConfig.ACKS_CONFIG), "all")
settings.put(StreamsConfig.producerPrefix(ProducerConfig.MAX_REQUEST_SIZE_CONFIG), 20.mb)

if (!isLocalRun)
  settings.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, "3")

My problem is, our deployments were working and suddenly everything start to have this error:

[kafka-producer-network-thread | myconsumergroup-1d0237ae-6caa-4cbd-aeaa-2154d2303b32-StreamThread-1-producer] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=myconsumergroup-1d0237ae-6caa-4cbd-aeaa-2154d2303b32-StreamThread-1-producer] Error while fetching metadata with correlation id 9 : {=INVALID_TOPIC_EXCEPTION}

I'm trying to find what is wrong, what is the problem with this configuration or something like that but no success. This start to fail even in local machine, what could be the causes?

Answer 1

The rubber duck attacks again...

Well the answer for the problem is, the configuration of the output topic was getting an empty string.

So the producer for the .to(outputTopic) was failing just after the mapValues