将Kafka主题的线索设置为-1

Question

I am consuming a kfka topic on the cluster using Spark streaming. I would restrat my streaming application and would like to read the topic mnessages again.

Is there a way to reset the kafka topic leads on the cluster?

when I describe the topics i get bellow description:

Topic:mahditest PartitionCount:3        ReplicationFactor:2     Configs:
        Topic: mahditest        Partition: 0    Leader: 1020    Replicas: 1020,1019     Isr: 1020,1019

but I need to resent leads to -1 for each topic so I can re-read the messages.

Answer 1

Depends on where you are storing the offsets but If you are storing them in Kafka using the new consumer API then in Kafka 0.11 you can reset any Kafka consumer's offsets (stored in __consumer_offsets topic) if you run

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --topic mahditest --group yourconsumer --reset-offsets --to-earliest --execute

You should shut down all consumers in the group before running the reset offsets command

The 0.11 version of this command should also work to reset offsets on a 0.10.2 cluster