[英]How does the offset for a topic work in Kafka (Kafka_net)
I have a basic producer app and a consumer app.我有一个基本的生产者应用程序和一个消费者应用程序。 if I run both and have both start consuming on their respective topics, I have a great working system.
如果我同时运行两者并且都开始关注各自的主题,那么我就有了一个很棒的工作系统。 My thought was that if I started the producer and sent a message that I would be able to then start the consumer and have it pick up that message.
我的想法是,如果我启动了生产者并发送了一条消息,那么我就可以启动消费者并让它接收该消息。 I was wrong.
我错了。
Unless both are up and running, I lose messages (or they do not get consumed).除非两者都启动并运行,否则我会丢失消息(或者它们不会被消耗)。
my consumer app looks like this for comsuming...我的消费者应用程序看起来像这样用于消费...
Uri uri = new Uri("http://localhost:9092");
KafkaOptions options = new KafkaOptions(uri);
BrokerRouter brokerRouter = new BrokerRouter(options);
Consumer consumer = new Consumer(new ConsumerOptions(receiveTopic, brokerRouter));
List<OffsetResponse> offset = consumer.GetTopicOffsetAsync(receiveTopic, 100000).Result;
IEnumerable<OffsetPosition> t = from x in offset select new OffsetPosition(x.PartitionId, x.Offsets.Max());
consumer.SetOffsetPosition(t.ToArray());
IEnumerable<KafkaNet.Protocol.Message> msgs = consumer.Consume();
foreach (KafkaNet.Protocol.Message msg in msgs)
{
do some stuff here based on the message received
}
unless I have the code between the lines, it starts at the beginning every time I start the application.除非我有两行之间的代码,否则每次启动应用程序时它都会从头开始。 What is the proper way to manage topic offsets so messages are consumed after a disconnect happens?
管理主题偏移以便在断开连接后使用消息的正确方法是什么?
If I run如果我跑
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message-reply-XXX consumer-property fetch-size=40000000 --from-beginning
I can see the messages, but when I connect my application to that topic, the consumer.Consume() does not pick up the messages it has not already seen.我可以看到消息,但是当我将我的应用程序连接到该主题时,consumer.Consume() 不会获取它尚未看到的消息。 I have tried this with and without runing the above bat file to see if that makes any difference.
我已经在运行和不运行上述 bat 文件的情况下进行了尝试,以查看是否有任何区别。 When I look at the consumer.SetOffsetPosition(t.ToArray()) call (t specifically) it shows that the offset is the count of all messages for the topic.
当我查看 consumer.SetOffsetPosition(t.ToArray()) 调用(特别是 t)时,它显示偏移量是该主题的所有消息的计数。
Please help,请帮忙,
Set auto.offset.reset
configuration in your ConsumerOptions
to earliest
.集
auto.offset.reset
在配置ConsumerOptions
来earliest
。 When the consumer group starts the consume messages, it will consume from the latest offset because the default value for auto.offset.reset
is latest.当消费者组开始消费消息时,它会从最近的偏移量开始消费,因为
auto.offset.reset
的默认值是最新的。
But I looked at kafka-net API now, it does not have a AutoOffsetReset
property, and it seems pretty insufficient with its configuration in consumers.但是我现在查看了kafka-net API,它没有
AutoOffsetReset
属性,而且它在消费者中的配置似乎很不够。 It also lacks documentation with method summaries.它还缺乏包含方法摘要的文档。
I would suggest you use Confluent .NET Kafka Nuget package because it is owned by Confluent itself.我建议您使用Confluent .NET Kafka Nuget 包,因为它归Confluent自己所有。
Also, why are calling GetTopicOffsets
and setting that offset back again in consumer.另外,为什么要调用
GetTopicOffsets
并在消费者中再次设置该偏移量。 I think when you configure your consumer, you should just start reading messages with Consume()
.我认为当你配置你的消费者时,你应该开始使用
Consume()
阅读消息。
Try this:尝试这个:
static void Main(string[] args)
{
var uri = new Uri("http://localhost:9092");
var kafkaOptions = new KafkaOptions(uri);
var brokerRouter = new BrokerRouter(kafkaOptions);
var consumerOptions = new ConsumerOptions(receivedTopic, brokerRouter);
var consumer = new Consumer(consumerOptions);
foreach (var msg in consumer.Consume())
{
var value = Encoding.UTF8.GetString(msg.Value);
// Process value here
}
}
In addition, enable logs in your KafkaOptions
and ConsumerOptions
, they will help you a lot:此外,在您的
KafkaOptions
和ConsumerOptions
启用日志,它们会对您有很大帮助:
var kafkaOptions = new KafkaOptions(uri)
{
Log = new ConsoleLog()
};
var consumerOptions = new ConsumerOptions(topic, brokerRouter)
{
Log = new ConsoleLog()
});
我切换到使用 Confluent 的 C# .NET 包,现在它可以工作了。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.