Do Kafka Streams have a timeout on how long processing can take?

Question

I'm using Kafka Streams that call external systems during map/foreach . Is there any timeout on how long map or foreach can take?

Are there any caveats to blocking for a long time (say hours)?

Answer 1

There is no timeout applied to map/foreach .

However, as Kafka Streams uses KafkaConsumer and KafkaProducer internally, all their timeouts apply (eg. max.poll.interval.ms ). You can of course configure them accordingly, but it's not recommended to do long blocking calls to external systems.

It would have many advantages, to load the data from you external service into a topic using Kafka Connect, and read this data as a KTable in your application a do a stream-table join instead of a map. This approach decouples your Streams application from the external system and thus makes the overall architecture more robust.