Spring Kafka Binder 没有收到任何消息但已连接到主题

Question

Hello I have been using Spring Kafka Binder as a consumer. Looking through the logs, I am able to connect to the topic although I am not sure why it is not processing any messages from the producer.

Any idea on what may be missing? Thank you!

POM

<dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-stream</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-stream-binder-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-config</artifactId>
    </dependency>

Application YML

cloud:
zookeeper:
  connect-string: port1.test.com:2181,port2.test.com:2181,port3.test.com:2181
stream:
  kafka:
    binder:
      brokers:
        - port1.test.com:6667
        - port2.test.com:6667
        - port3.test.com:6667
      auto-create-topics: false
      auto-add-partitions: false
      jaas:
        controlFlag: REQUIRED
        loginModule: com.sun.security.auth.module.Krb5LoginModule
        options:
          useKeyTab: true
          storeKey: true
          serviceName: kafka
          # Change location to your local location
          keyTab: C:\\Users\\src\\main\\resources\\kafka\\kafka_user.keytab
          principal: kafka_user@TEST.COM
          debug: true
      configuration:
        security:
          protocol: SASL_PLAINTEXT
  bindings:
    stream-input:
      binder: kafka
      destination: TOPIC
      group: service-dev
security:
  krb5conf:
    # Change location to your local location
    location: C:\\Users\\src\\main\\resources\\kafka\\krb5nonprod.conf

Consumer class

public interface EventConsumer {

@Input("stream-input")
SubscribableChannel consumeMessage();
}

Listener class

@Slf4j
@Component
@RequiredArgsConstructor
@EnableBinding(EventConsumer.class)
public class EventListener {

@StreamListener(target = "stream-input")
public void processMessage(Object msg) {

Logs

Started Application in 75.471 seconds (JVM running for 184.663)
2021-09-29 19:45:01.342  INFO 30340 --- [container-0-C-1] org.apache.kafka.clients.Metadata        
: [Consumer clientId=consumer-2, groupId=service-dev] Cluster ID: qa_IFa70SravzxvdcDhHA
2021-09-29 19:45:01.390  INFO 30340 --- [container-0-C-1] 
o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=service-dev] 
Discovered group coordinator port1.test.com:6667 (id: 2147482644 rack: null)
2021-09-29 19:45:01.399  INFO 30340 --- [container-0-C-1] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=service-dev] 
Revoking previously assigned partitions []
2021-09-29 19:45:01.400  INFO 30340 --- [container-0-C-1] 
o.s.c.s.b.k.KafkaMessageChannelBinder$1  : service-dev: partitions revoked: []
2021-09-29 19:45:01.401  INFO 30340 --- [container-0-C-1] 
o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=service-dev] 
(Re-)joining group
2021-09-29 19:45:01.854  INFO 30340 --- [container-0-C-1] 
o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=service-dev] 
(Re-)joining group
2021-09-29 19:45:04.387  INFO 30340 --- [container-0-C-1] 
o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=service-dev] 
Successfully joined group with generation 36
2021-09-29 19:45:04.400  INFO 30340 --- [container-0-C-1] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=service-dev] 
Setting newly assigned partitions: TOPIC-0
2021-09-29 19:45:04.481  INFO 30340 --- [container-0-C-1] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=service-dev] 
Setting offset for partition TOPIC-0 to the committed offset FetchPosition{offset=1076, 
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=port1.test.com:6667 (id: 1003 
rack: /default-rack), epoch=2}}
2021-09-29 19:45:04.557  INFO 30340 --- [container-0-C-1] 
o.s.c.s.b.k.KafkaMessageChannelBinder$1  : service-dev: partitions assigned: [TOPIC-0]

As you can see from the logs, it was able to connect to the topic. Although I am not sure why I am not receiving any messages from the producer. Is it because of the revoking of the partitions? Does it have something to do on why I am not receiving any messages? The producer comes from a 3rd party, does he needs to do something in order for me to receive messages? It is clear that I am able to connect to the topic. Thank you!

Answer 1

The Log looks good; it implies there are no more records to read after the current committed offset for this group

>Setting offset for partition TOPIC-0 to the committed offset FetchPosition{offset=1076 ...

If you want to re-read the entire topic you can set the resetOffsets kafka consumer binding property, or change the group to one that doesn't have a committed offset.

https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.4/reference/html/spring-cloud-stream-binder-kafka.html#reset-offsets