Getting Timed out reading socket error for kafka cluster setup

Question

I want to setup a kafka cluster type setup for three similar application having same queues like AppA -> {TopicX, TopicY, TopicZ}, AppB -> {TopicX, TopicZ}, AppC -> {TopicX, TopicY}. Producer and Consumer will be App specific. I setup kafka cluster with three brokers having partition 1,2,3 in three different config files with different ports. Then start kafka server ( cluster )

I am using kafka php wrapper by http://github.com/nmred/kafka-php

So I used Producer code for App A like

       $producer->setRequireAck(-1);
       $producer->setMessages("TopicX", 0, array(json_encode($this->data)));
       $producer->send();

AND used Producer code for App B like

       $producer->setRequireAck(-1);
       $producer->setMessages("TopicX", 1, array(json_encode($this->data)));
       $producer->send();

And So On.

Then I made my Consumer scripts for three apps like

        $queues = array("TopicX", "TopicY", "TopicZ");
        while(true) {
            foreach($queues as $queue) {
                $consumer = \Kafka\Consumer::getInstance('localhost:2181');
                $consumer->setGroup('testgroup');
                $consumer->setPartition($queue, 0);
                $result = $consumer->fetch();
           }
        }

But when I try to execute consumer script for any App I get error like

"Timed out reading socket while reading 750437 bytes with 750323 bytes to go"

I just don't know How I can fix this issue I tried to modify some kafka config parameters like

 zookeeper.connection.timeout.ms=24000         # Initially 6000
 replica.socket.timeout.ms=15000                      # Not exists in default file

but that not worked.

Answer 1

You're actually destroying the Kafka consumer by declaring it into your foreach cycle, move it out of the cycle, use the cycle to set the partitions and then fetch the results and discriminate between the sources:

$queues = array("TopicX", "TopicY", "TopicZ");
$consumer = \Kafka\Consumer::getInstance('localhost:2181');
$consumer->setGroup('testgroup');

foreach($queues as $queue) {
  $consumer->setPartition($queue, 0);
}

while(true) {
  $result = $consumer->fetch();
  foreach ($result as $topicName => $topic) {
    foreach ($topic as $partId => $partition) {
      var_dump($partition->getHighOffset());
      foreach ($partition as $message) {
        var_dump((string)$message);
      }
    }
  }
}

See the github README of the kafka php project to see an example.