Kinesis as producer in Spring Boot Reactive Stream API

Question

I'm trying to build a small Spring Boot Reactive API. The API should let the users subscribe to some data, returned as SSE.

The data is located on a Kinesis Topic.

Creating the Reactive API, and the StreamListener to Kinesis is fairly easy - but can I combine these, so the Kinesis Topic are used as a producer for the event stream used by my data service.

The code looks more or less like this

//Kinesis binding, with listenerMode: rawRecords
@EnableBinding(Sink.class)
public class KinesisStreamListener {

  @StreamListener(value = Sink.INPUT)
  public void logger(List<Record> payload) throws Exception {

  }
}

@RestController
@RequestMapping("/data")
public class DataResource {

  @Autowired
  DataService service;

  @GetMapping(produces = {MediaType.TEXT_EVENT_STREAM_VALUE, MediaType.APPLICATION_STREAM_JSON_VALUE})
  public Flux<EventObject> getData() {
    return service.getData();
  }
}

@Component
public class DataService {

  Flux<EventObject> getData() {
    Flux<Long> interval = Flux.interval(Duration.ofMillis(1000));
    Flux<EventObject> dataFlux = Flux.fromStream(Stream.generate(() -> ???
            ));
      return dataFlux.zip(interval, dataFlux).map(Tuple2::getT2);
  }
}

Answer 1

Here is a sample how I would do that: https://github.com/artembilan/sandbox/tree/master/cloud-stream-kinesis-to-webflux .

Once we agree about details and some improvements it can go to the official Spring Cloud Stream Samples repository: https://github.com/spring-cloud/spring-cloud-stream-samples

The main idea is to reuse the same Flux provided by the @StreamListener via Spring Cloud Stream Reactive Support. This is is already a FluxPublish , so any new SSE connections will work as a plain Reactive subscribers.

There are a couple tricks to count with:

1. For the listenerMode: rawRecords , we also need to configure a contentType: application/octet-stream to avoid any conversion attempts when Binder sends a message to the Sink.INPUT channel.
2. Since listenerMode: rawRecords returns a List<Record> our Flux in the @StreamListener method should expect exactly this type, but not a plain Record .

Both concerns are considered as a Framework improvements.

So, let us now how it looks and works for you.