[英]C# Unable to consume message on Kafka topic?
I've been looking through several examples of the Confluent.Kafka client ( https://github.com/confluentinc/confluent-kafka-dotnet/ ), and whilst I can successfully get a producer to push a message into Kafka, I'm unable to pull any messages back down with consumers. 我一直在查看Confluent.Kafka客户端的几个例子( https://github.com/confluentinc/confluent-kafka-dotnet/ ),虽然我可以成功让生产者将消息发送到Kafka,但我我无法将任何消息拉回消费者手中。
Through the UI I can see the topic is created and messages are going into this topic (there are currently 10 partitions and 3 messages), but my consumer always reports "end of partition", without any consumption of any message (the 3 remain on the topic and "OnMessage" never fires). 通过UI,我可以看到主题已创建,消息将进入此主题(目前有10个分区和3个消息),但我的消费者总是报告“分区结束”,而不消耗任何消息(3保留在主题和“OnMessage”永远不会激发)。
However the consumer is definitely accessing the topics, and can see 3 messages on one of the partitions: 但是,消费者肯定会访问这些主题,并且可以在其中一个分区上看到3条消息:
end of partition: dotnet-test-topic [6] @3 分区结束:dotnet-test-topic [6] @ 3
It just doesn't consume the message and trigger OnMessage(). 它只是不消耗消息并触发OnMessage()。 Any ideas?
有任何想法吗?
var conf = new Dictionary<string, object>
{
{ "group.id", Guid.NewGuid().ToString() },
{ "bootstrap.servers", "mykafkacluster:9094" },
{ "sasl.mechanisms", "SCRAM-SHA-256" },
{ "security.protocol", "SASL_SSL" },
{ "sasl.username", "myuser" },
{ "sasl.password", "mypass" }
};
using (var producer = new Producer<string, string>(conf, new StringSerializer(Encoding.UTF8), new StringSerializer(Encoding.UTF8)))
{
producer.ProduceAsync("dotnet-test-topic", "some key", "some value")
.ContinueWith(result =>
{
var msg = result.Result;
if (msg.Error.Code != ErrorCode.NoError)
{
Console.WriteLine($"failed to deliver message: {msg.Error.Reason}");
}
else
{
Console.WriteLine($"delivered to: {result.Result.TopicPartitionOffset}");
}
});
producer.Flush(TimeSpan.FromSeconds(10));
}
using (var consumer = new Consumer<string, string>(conf, new StringDeserializer(Encoding.UTF8), new StringDeserializer(Encoding.UTF8)))
{
consumer.Subscribe("dotnet-test-topic");
consumer.OnConsumeError += (_, err)
=> Console.WriteLine($"consume error: {err.Error.Reason}");
consumer.OnMessage += (_, msg)
=> Console.WriteLine($"consumed: {msg.Value}");
consumer.OnPartitionEOF += (_, tpo)
=> Console.WriteLine($"end of partition: {tpo}");
while (true)
{
consumer.Poll(TimeSpan.FromMilliseconds(100));
}
}
Looks like the OnMessage event won't fire without the following config provided: 看起来如果没有提供以下配置,OnMessage事件将不会触发:
{ "auto.offset.reset", "smallest" }
With this added, I was able to read the messages on the topic. 通过这个添加,我能够阅读有关该主题的消息。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.