简体   繁体   English

我未订阅的主题的Kafka错误

[英]Kafka Error For Topics I'm Not Subscribed To

I saw a kafka warning for topics my current instance is not subscribed to. 我看到有关当前实例未订阅的主题的kafka警告。 We use the same kafka broker, and each dev has his/her own topic. 我们使用相同的kafka经纪人,每个开发人员都有自己的主题。 However we all use the same default groupID/clientid specified in a properties file. 但是,我们都使用在属性文件中指定的相同的默认groupID / clientid。

2017-06-27T11:48:35.20-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,195 WARN [NetworkClient] - [kafka-coordinator-heartbeat-thread | ingestion-matching-kafka-consumer-group] - Error while fetching metadata with correlation id 154274 : {to-process-yi=UNKNOWN_TOPIC_OR_PARTITION, to-process-shawn=UNKNOWN_TOPIC_OR_PARTITION}
2017-06-27T11:48:35.34-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,339 INFO [PdfXmlMatcherTask] - [scheduler-1] -
2017-06-27T11:48:35.34-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,340 INFO [PdfXmlMatcherTask] - [scheduler-1] - Checking messages...
2017-06-27T11:48:35.41-0400 [APP/PROC/WEB/0]OUT 2017-06-27 15:48:35,412 WARN [NetworkClient] - [scheduler-1] - Error while fetching metadata with correlation id 154275 : {to-process-yi=UNKNOWN_TOPIC_OR_PARTITION, to-process-shawn=UNKNOWN_TOPIC_OR_PARTITION}

Is this expected? 这是预期的吗? Do I need separate group/client IDs even though each person has their own topic? 即使每个人都有自己的主题,我是否需要单独的组/客户ID?

Secondly I'm not quite sure what this error means: the heart-beat coordinator thread couldn't fetch metadata for 2 topics, but these topics exist, it's just that the consumers for those topics aren't running. 其次,我不太确定此错误的含义:心跳协调器线程无法获取2个主题的元数据,但是这些主题存在,只是这些主题的使用者未运行。

client.id - is useful for tracing requests, should be unique to each client (though not mandatory). client.id-用于跟踪请求,对于每个客户端应该唯一(尽管不是强制性的)。

groupid - this applies only to consumers. groupid-仅适用于消费者。 By your short description you probably want each consumer in its own group. 通过简短的描述,您可能希望每个消费者都属于自己的组。

I suggest you read what ConsumerGroups are for : "Kafka scales topic consumption by distributing partitions among a consumer group, which is a set of consumers sharing a common group identifier. " https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/ 我建议您阅读ConsumerGroups的用途:“ Kafka通过在消费者组之间分配分区来扩展主题的使用,消费者组是一组共享公用组标识符的消费者。” https://www.confluent.io/blog/tutorial-getting开始使用新的Apache Kafka-0-9消费者客户端/

UNKNOWN_TOPIC_OR_PARTITION is an error that the broker returns exactly when a topic or partition does not exist. UNKNOWN_TOPIC_OR_PARTITION是当主题或分区不存在时,代理完全返回的错误。 it's an erro that the Kafka client treats as retriable. Kafka客户将其视为可重试是一种错误。

If you think that the topic exists, please check twice. 如果您认为该主题存在,请检查两次。 It could also be the case that the topic exists but a producer or consumer is targeting directly a non-existing partition of an existing topic. 也可能是该主题存在,但生产者或消费者直接将现有主题的不存在分区作为目标。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 卡夫卡喷口错误“消费者未订阅任何主题或未分配任何分区” - Kafka spout error “Consumer is not subscribed to any topics or assigned any partitions ” 我可以在单个 mule 应用程序中拥有两个 kafka 消费者吗? 都订阅了不同的主题 - Can i have two kafka consumer in single mule app? both subscribed to different topics 订阅3个以上主题时,Kafka消费者未收到消息 - Kafka Consumer Not Receiving Messages When Subscribed to 3+ Topics Kafka 对通过 topicPattern 订阅多个主题的消费者进行非阻塞重试 - Kafka non blocking retry on consumer subscribed to multiple topics by topicPattern Kafka 消费者订阅了同一消费者组中的不同主题 - Kafka Consumers subscribed to different topics in same consumer group 如果 Kafka Consumer 订阅了多个主题,它是否会以循环方式接收来自不同主题的消息? - If a Kafka Consumer is subscribed to multiple topics, does it receive messages in a round robin style from the different topics? kafka python - 找到消费者尚未订阅的新 kafka 主题的正确方法是什么? - kafka python - What is the right way to find new kafka topics that a consumer has not been yet subscribed to? 卡夫卡用例的错误主题 - kafka use-case for error topics 创建数千个 Kafka 主题不会出错 - Creating thousands of Kafka topics stops without error 如何查看和删除Kafka主题 - How do I view and delete Kafka topics
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM