Reading all messages from a Kafka topic

Question

I have the following use case:

I have two Kafka topics, one is meant to be used a stream of incoming messages to be processed, the other is meant as store of records that is meant to be used as a bootstrap to the initial state of the application.

Is there a way to do the following:

1. Read all messages from a Kafka topic when the application starts up and store all ConsumerRecord in memory from the topic that is meant to bootstrap the application to its initial state
2. Only after all messages have been read allow the ConsumerRecord from the stream topic to be processed
3. As there may be additional records on the state topic to incorporate them into the application's state when the application is running without having to restart the application.

Thanks!

Answer 1

Start your bootstrap consumer first.

Read the other topic till a particular offset is reached or (if you want the end, you can read as long as there is no polled records available [this is not the best way!]). If you want to start at particular offset every-time you have to use a seek. Also use a unique consumer group id for this since you want to all the records. You might want to handle the rebalance case appropriately.

Then close that consumer and start the other stream consumer and process the data. Using Ktables with Kafka streams might be better, but I am not familiar with it.