spring-cloud-stream kafka offline consumer messages lost

Question

I've been playing with spring-cloud-stream (1.0.0.BUILD-SNAPSHOT with kafka binder) and noticed that with the consumer offline, any messages sent are lost. When I start the consumer it does not process a backlog of requests that were sent to kafka. Is that intentional?

Answer 1

We definitely need to improve our documentation but here are some pointers in the mean time.

If you want the consumer to process messages produced while they were stopped, you need to specify a consumer group name, eg spring.cloud.stream.bindings.<bindingName>.group=foo . When a consumer group is specified, the application with start at either a) the latest unconsumed message if a client with the same consumer group has run already (ie we recorded consumed offsets for that consumer) or b) the value specified by spring.cloud.stream.binder.kafka.start-offset (which can be earliest or latest , representing the start or the end of the topic). So restarting consumers that preserve the consumer group will consume from where they left, and new consumers will start according to the start option. If a group is not specified, then the consumer will be considered 'anonymous', and only be interested in messages produced after it has started, therefore it will always start at the end of the partition set.

If you wanted to circumvent the already saved value, then you can use the spring.cloud.stream.binder.kafka.reset-offets=true , which will cause the client to reset the saved offsets and start at the value indicated by spring.cloud.stream.binder.kafka.start-offset .

This reflects the behaviour expected by (and supported with) 0.8.2. We will update things accordingly once we upgrate to 0.9.

Answer 2

Yes; the default is to start listening from the end of the topic.

Use

spring:
  cloud:
    stream:
      binder:
        kafka:
          start-offset: earliest