消費者在 Apache Kafka 中消費消息的延遲

Question

我正在使用 Kafka 0.8.0 並試圖實現下面提到的場景。

JCA API（充當生產者並向其發送數據）-----> 消費者------> HBase

一旦我使用 JCA Client 獲取數據，我就會將每條消息發送給消費者。 例如，一旦生產者發送消息 no.1 ，我想從消費者那里獲取相同的信息並“放入”HBase。 但是我的消費者在一些隨機 n 消息之后開始獲取消息。 我想讓生產者和消費者同步，以便他們開始一起工作。

我用過了：

1個經紀人

1 個主題

1 單個生產者和高級消費者

誰能建議我需要做什么才能達到同樣的目標？

編輯：

添加一些相關的代碼片段。

消費者.java

public class Consumer extends Thread {
    private final ConsumerConnector consumer;
    private final String topic;
    PrintWriter pw = null;
    int t = 0;
    StringDecoder kd = new StringDecoder(null);
    Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
    Map<String, List<KafkaStream<String, Signal>>> consumerMap;
    KafkaStream<String, Signal> stream;
    ConsumerIterator<String, Signal> it;

    public Consumer(String topic) {
        consumer = kafka.consumer.Consumer.createJavaConsumerConnector(createConsumerConfig());

        this.topic = topic;
        topicCountMap.put(topic, new Integer(1));
        consumerMap = consumer.createMessageStreams(topicCountMap, kd, new Serializer(
                new VerifiableProperties()));
        stream = consumerMap.get(topic).get(0);
        it = stream.iterator();

    }

    private static ConsumerConfig createConsumerConfig() {
        Properties props = new Properties();
        props.put("zookeeper.connect", KafkaProperties.zkConnect);
        props.put("group.id", KafkaProperties.groupId);
        props.put("zookeeper.session.timeout.ms", "400");
        props.put("zookeeper.sync.time.ms", "200");
        props.put("auto.commit.interval.ms", "1000");
        props.put("fetch.size", "1024");

        return new ConsumerConfig(props);

    }

    synchronized public void run() {

        while (it.hasNext()) {
            t = (it.next().message()).getChannelid();
            System.out.println("In Consumer received msg" + t);
        }
    }
}

生產者.java

public class Producer {
    public final kafka.javaapi.producer.Producer<String, Signal> producer;
    private final String topic;
    private final Properties props = new Properties();

    public Producer(String topic)
    {
        props.put("serializer.class", "org.bigdata.kafka.Serializer");
        props.put("key.serializer.class", "kafka.serializer.StringEncoder");
        props.put("metadata.broker.list", "localhost:9092");
        // Use random partitioner. Don't need the key type. Just set it to Integer.
        // The message is of type userdefined Object .
        producer = new kafka.javaapi.producer.Producer<String,Signal(newProducerConfig(props));
        this.topic = topic;
    }
}

KafkaProperties.java

public interface KafkaProperties {
    final static String zkConnect = "127.0.0.1:2181";
    final static String groupId = "group1";
    final static String topic = "test00";
    final static String kafkaServerURL = "localhost";
    final static int kafkaServerPort = 9092;
    final static int kafkaProducerBufferSize = 64 * 1024;
    final static int connectionTimeOut = 100000;
    final static int reconnectInterval = 10000;
    final static String clientId = "SimpleConsumerDemoClient";
}

這就是消費者對前 10 條消息的行為方式，它不會對消費者收到的消息進行系統輸出，但從第 11 條消息開始，它開始正常運行。

     producer sending msg1

     producer sending msg2

     producer sending msg3

     producer sending msg4

     producer sending msg5

     producer sending msg6

     producer sending msg7

     producer sending msg8

     producer sending msg9

     producer sending msg10

     producer sending msg11

     producer sending msg12
     In Consumer received msg12

     producer sending msg13
     In Consumer received msg13

     producer sending msg14
     In Consumer received msg14

     producer sending msg15
     In Consumer received msg15

     producer sending msg16
     In Consumer received msg16

     producer sending msg17
     In Consumer received msg17

     producer sending msg18
     In Consumer received msg18

     producer sending msg19
     In Consumer received msg19

     producer sending msg20
     In Consumer received msg20

     producer sending msg21
     In Consumer received msg21

編輯：在生產者向消費者發送消息的地方添加偵聽器功能。 而且我使用的默認生產者配置沒有覆蓋它

public synchronized void onValueChanged(final MonitorEvent event_) {


    // Get the value from the DBR
    try {
        final DBR dbr = event_.getDBR();

        final String[] val = (String[]) dbr.getValue();

        producer1.producer.send(new KeyedMessage<String, Signal>         
                    (KafkaProperties.topic,new Signal(messageNo)));
        System.out.println("producer sending msg"+messageNo);

        messageNo++;


    } catch (Exception ex) {
        ex.printStackTrace();
    }
}

Answer 1

1. 嘗試將props.put("request.required.acks", "1")到生產者配置中。 默認情況下，生產者不等待確認並且不保證消息傳遞。 因此，如果您在測試之前啟動 broker，生產者可能會在 broker 完全初始化之前開始發送消息，並且前幾條消息可能會丟失。

2. 嘗試將props.put("auto.offset.reset", "smallest")到消費者配置中。 它等於 kafka-console-consumer.sh 的 --from --from-beginning選項。 如果您的消費者晚於生產者啟動並且 Zookeeper 中沒有保存偏移數據，那么默認情況下它將開始僅使用新消息（請參閱文檔中的消費者配置）。

Answer 2

一種可能性是卡夫卡滯后。 可能是消費者因分區過多而過載。 或者處理每條消息的成本非常高。