我如何從 Spring Cloud Stream Kafka Binder 中的偏移量獲取消息？

Question

我能夠連接到該主題，但我不確定如何從該主題獲取消息。 這是顯示我有 1076 條關於該主題的記錄的日志：

日志

2021-10-05 11:10:09.053  INFO 17364 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder    : Partitions revoked: []
2021-10-05 11:10:09.054  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] (Re-)joining group
2021-10-05 11:10:10.063  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] (Re-)joining group
2021-10-05 11:10:13.189  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Successfully joined group with generation 1
2021-10-05 11:10:13.205  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Setting newly assigned partitions: MY_TOPIC-0
2021-10-05 11:10:13.278  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Found no committed offset for partition MY_TOPIC-0
2021-10-05 11:10:13.664  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-2, groupId=latest] Resetting offset for partition MY_TOPIC-0 to offset 1076.
2021-10-05 11:10:13.730  INFO 17364 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder    : Partitions assigned: [MY_TOPIC-0]
2021-10-05 11:10:13.731  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-2, groupId=latest] Seeking to EARLIEST offset of partition MY_TOPIC-0
2021-10-05 11:10:13.787  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-2, groupId=latest] Resetting offset for partition MY_TOPIC-0 to offset 1076.
2021-10-05 11:46:44.345  INFO 17364 --- [thread | latest] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Group coordinator test.kafka.com:6667 (id: 2246481044 rack: null) is unavailable or invalid, will attempt rediscovery
2021-10-05 11:46:51.625  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Discovered group coordinator test.kafka.com:6667 (id: 2147482644 rack: null)
2021-10-05 11:46:56.514  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Attempt to heartbeat failed for since member id consumer-2-386b3e7b-b8a1-48c5-9gd3-5e587e4237ad is not valid.
2021-10-05 11:46:56.516  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Revoking previously assigned partitions [MY_TOPIC-0]
2021-10-05 11:46:56.516  INFO 17364 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder    : Partitions revoked: [MY_TOPIC-0]
2021-10-05 11:46:56.516  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] (Re-)joining group
2021-10-05 11:46:56.572  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] (Re-)joining group
2021-10-05 11:46:59.687  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Successfully joined group with generation 3
2021-10-05 11:46:59.688  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Setting newly assigned partitions: MY_TOPIC-0
2021-10-05 11:46:59.749  INFO 17364 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=latest] Setting offset for partition MY_TOPIC-0 to the committed offset FetchPosition{offset=1076, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=test.kafka.com:6667 (id: 1003 rack: /default-rack), epoch=2}}
2021-10-05 11:46:59.812  INFO 17364 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder    : Partitions assigned: [MY_TOPIC-0]

消費者 Class

public interface EventConsumer {

    @Input("my-group-id")
    SubscribableChannel consumeMessage();

}

聽眾 Class

@Slf4j
@Component
@RequiredArgsConstructor
@EnableBinding(EventConsumer.class)
public class EventListener {

     @StreamListener(target = "my-group-id")
     public void processMessage(Object msg) {
         log.info("*** MESSAGE: ***", msg);
         **do something**
         **save messages**
     }
}

應用.yml

kafka:
    consumer:
      properties:
        max.poll.interval.ms: 3600000
      max-poll-records: 10
  cloud:
    zookeeper:
      connect-string: test.kafka.com:2181,test.kafka.com:2181,test.kafka.com:2181
    stream:
      kafka:
        bindings:
          my-group-id:
            consumer:
              autoCommitOffset: false
        binder:
          brokers:
            - test.kafka.com:6667
            - test.kafka.com:6667
            - test.kafka.com:6667
          auto-create-topics: false
          auto-add-partitions: false
          jaas:
            controlFlag: REQUIRED
            loginModule: com.sun.security.auth.module.Krb5LoginModule
            options:
              useKeyTab: true
              storeKey: true
              serviceName: kafka
              keyTab: C:\\files\\user.keytab
              principal: user@test.com
              debug: true
          configuration:
            security:
              protocol: SASL_PLAINTEXT
      bindings:
        my-group-id:
          binder: kafka
          destination: MY_TOPIC
          group: test-kafka-service
  servlet:
    multipart:
      max-file-size: 50MB
      max-request-size: 50MB
spring.cloud.stream.kafka.bindings.my-group-id.consumer.resetOffsets: true
spring.cloud.stream.kafka.streams.binder.configuration.commit.interval.ms: 300000

閱讀日志后，它甚至沒有 go 到我的監聽器 class，我在其中放置了一個記錄器。 對此有什么想法嗎？

Answer 1

要讓@StreamListener在日志的開頭再次啟動，請在綁定上配置一個group （以便將 auto.offset.reset 設置為earliest並將resetOffsets設置為true 。

spring.cloud.stream.kafka.bindings.my-group-id.consumer.resetOffsets=true

https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.4/reference/html/spring-cloud-stream-binder-kafka.html#kafka-consumer-properties

重置偏移量

是否將消費者的偏移量重置為 startOffset 提供的值。 如果提供了KafkaBindingRebalanceListener ，則必須為 false； 請參閱使用 KafkaBindingRebalanceListener。 有關此屬性的更多信息，請參閱重置偏移量。

默認值：假。

https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.4/reference/html/spring-cloud-stream-binder-kafka.html#reset-offsets

編輯

這對我來說可以：

@SpringBootApplication
public class So69432739Application {

    public static void main(String[] args) {
        SpringApplication.run(So69432739Application.class, args);
    }

    @Bean
    public Consumer<String> input() {
        return System.out::println;
    }

    @Bean
    ApplicationRunner runner(KafkaOperations<byte[], byte[]> ops) {
        return args -> {
            ops.send("input-in-0", "one".getBytes());
            ops.send("input-in-0", "two".getBytes());
            ops.send("input-in-0", "three".getBytes());
        };
    }

}

spring.cloud.stream.bindings.input-in-0.group=grp
spring.cloud.stream.kafka.bindings.input-in-0.consumer.reset-offsets=true

我第二次運行它：

one
two
three
one
two
three

Setting offset for partition input-in-0-0 to the committed offset FetchPosition{offset=3, ...
Resetting offset for partition input-in-0-0 to position FetchPosition{offset=0, ...

請注意，我使用的是較新的功能樣式（不推薦使用@StreamListener ）； 盡管這對該功能沒有影響。

編輯2

你不能那樣混合 propertie 和 YAML； 我復制了你的 YAML（注釋掉了一些我不需要的）並且它工作正常......

spring:
  kafka:
    consumer:
      properties:
        max.poll.interval.ms: 3600000
      max-poll-records: 10
  cloud:
    stream:
      kafka:
        bindings:
          my-group-id:
            consumer:
              autoCommitOffset: false
              reset-offsets: true
#        binder:
#          brokers:
#            - test.kafka.com:6667
#            - test.kafka.com:6667
#            - test.kafka.com:6667
#          auto-create-topics: false
#          auto-add-partitions: false
#          jaas:
#            controlFlag: REQUIRED
#            loginModule: com.sun.security.auth.module.Krb5LoginModule
#            options:
#              useKeyTab: true
#              storeKey: true
#              serviceName: kafka
#              keyTab: C:\\files\\user.keytab
#              principal: user@test.com
#              debug: true
#          configuration:
#            security:
#              protocol: SASL_PLAINTEXT
      bindings:
        input-int-0:
          binder: kafka
          destination: input-in-0
          group: test-kafka-service
#  servlet:
#    multipart:
#      max-file-size: 50MB
#      max-request-size: 50MB

Setting offset for partition input-in-0-0 to the committed offset FetchPosition{offset=6 ...
Seeking to EARLIEST offset of partition input-in-0-0
Resetting offset for partition input-in-0-0 to position FetchPosition{offset=0 ...