如何在 Kafka Streams 中使用 Futures

Question

When using the org.apache.kafka.streams.KafkaStreams library in Scala, I have been trying to read in an inputStream, pass that information over to a method: validateAll(infoToValidate) that returns a Future, resolve that and then send to an output stream.

Example:

builder.stream[String, Object](REQUEST_TOPIC)
      .mapValues(v => ValidateFormat.from(v.asInstanceOf[GenericRecord]))
      .mapValues(infoToValidate => {
        SuccessFailFormat.to(validateAll(infoToValidate))
      })

Is there any documentation on performing this? I have looked into filter() and transform() but still not sure how to deal with Futures in KStreams.

Answer 1

The answer depends whether you need to preserve the original order of your messages. If yes, then you will have to block in one way or the other. For example:

val duration = 10 seconds // whatever your timeout should be or Duration.Inf
sourceStream
  .mapValues(x => Await.result(validate(x), duration))
  .to(outputTopic)

If however the order is not important, you can simply use a Kafka producer:

sourceStream
  .mapValues(x => validate(x)) // now you have KStream[.., Future[...]]
  .foreach { future =>
    future.foreach { item =>
      val record = new ProducerRecord(outputTopic, key, item)
      producer.send(record) // provided you have the implicit serializer
    }
  }