Project Reactor Kafka: Perform action at the end of Flux without blocking

Question

I am working on an application that is using project-reactor Kafka APIs to connect reactively to Kafka-brokers . The use-case is that there is an input topic which contains file-paths to the files for processing. The application reads each file, processes it, creates a flux of the processed messages and pushes it to the output topic. The requirement is that the file must be deleted once it has been processed and processed messages should be pushed to the output topic. So, the delete action must be performed after each file has been processed and the flux of the message pushed to the output topic.

public Flux<?> flux() {
   return KafkaReceiver
   .create(receiverOptions(Collections.singleton(sourceTopic)))
   .receive()
   .flatMap(m -> transform(m.value()).map(x -> SenderRecord.create(x, 
   m.receiverOffset())))
   .as(sender::send)
   .doOnNext(m -> {
   m.correlationMetadata().acknowledge();
   deleteFile(path);
}).doOnCancel(() -> close());

}

*The transform() method initiates the file processing in the file path(m.value()) and returns a flux of messages.

The problem is that the file is deleted even before all the messages is pushed to output topic. Therefore, in case of a failure, on re-try the original file is not available.

Answer 1

Since it seems the path variable is accessible in the whole pipeline (method input parameter?), you could delete the file within a separate doFinally . You would need to filter for onComplete or cancel SignalType , because you don't want to delete the file in case of a failure.

Another option would be doOnComplete if you're not interested in deleting the file upon cancellation.