简体   繁体   English

Java 中的 Apache Kafka 消费者程序

[英]Apache Kafka consumer program in Java

I am very new to Apache Kafka.我对 Apache Kafka 很陌生。 I am given a task to write a consumer that will consume messages from topic(s).我的任务是编写一个消费者,该消费者将消费来自主题的消息。 Below are my queries:以下是我的疑问:

  1. Should I tell my consumer Java program by providing the topic name explicitly (by passing as argument to main method) or is there any way by which it automatically discovers the topics?我应该通过显式提供主题名称(通过作为参数传递给 main 方法)来告诉我的消费者 Java 程序,还是有什么方法可以自动发现主题?

  2. I read somewhere that consumer program should be multi-threaded and after going through this link https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example , I think that it is creating number of threads for reading from single topic.我在某处读到消​​费者程序应该是多线程的,在通过此链接https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example 后,我认为它正在创建用于阅读的线程数从单个主题。

    I thought what multi-threaded means is that my consumer program should spawn as many threads as topics are.我认为多线程意味着我的消费者程序应该产生与主题一样多的线程。 If I have to read messages from, say, 3 topics, then do I need to start 3 consumer programs one for each topic?如果我必须从 3 个主题读取消息,那么我是否需要为每个主题启动 3 个消费者程序? Can't I do the same in single program?我不能在单个程序中做同样的事情吗? Please clarify.请说清楚。

  3. I read there are two types of consumer programs: Simple and High Level.我读到有两种类型的消费者程序:简单和高级。 I don't know which one to write.不知道写哪一篇。

How can I get started on this?我该如何开始呢?

1) Getting list of topic from the Kafka cluster could be a tricky job as there are no such direct api available for this .. the best bet is to query zookeeper as Kafka use it to maintain all of its states, this links explains about the zookeeper data structure . 1) 从 Kafka 集群获取主题列表可能是一项棘手的工作,因为没有这样的直接 api 可用于此 .. 最好的办法是查询 Zookeeper,因为 Kafka 使用它来维护其所有状态, 此链接解释了动物园管理员数据结构 You can read this link for a similar discussion您可以阅读链接以进行类似的讨论

2) the example assumes that you will use it to fetch data from a given topic, you might try to create a list of topics and loop through them to see how it goes, some thing like 2)该示例假设您将使用它从给定主题中获取数据,您可能会尝试创建一个主题列表并遍历它们以查看它是如何进行的,例如

     for(String topic : topicList) {
         new ConsumerGroupExample(zooKeeper, groupId, topic).run(threads);
     }

but not sure about the exact outcome of this, would be great if you could share your finding但不确定这个的确切结果,如果你能分享你的发现会很棒

3) Depends how much control you want over the topics based on your use cases. 3)取决于您希望根据您的用例对主题进行多少控制。 The high level does a lot of lot of bookkeeping and error handling but does not allow fine grain access over topic such as reading same message multiple times, consuming particular subset of messages etc where as all these functionality is provided by the simple consumer but in that case you need to handle cases like topic offset management, broker and leader detection etc.高层做了大量的簿记和错误处理,但不允许对主题进行细粒度访问,例如多次读取相同的消息、使用特定的消息子集等,因为所有这些功能都是由简单的消费者提供的,但在那个如果您需要处理主题偏移管理、代理和领导者检测等情况。
You should take a detail look in their consmer documentation for a much in-dept understanding of the same.您应该详细查看他们的消费者文档,以便深入了解相同内容。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM