运行Kafka性能流量时出现错误“此服务器不是该主题分区的领导者”

Question

Update Aug 15, 2018 I executed strace to monitor system call mprotect , and found it was blocked for several seconds indeed.

     strace -f -e trace=mprotect,mmap,munmap -T -t -p `pidof java` 2>&1 |tee mp1.txt

     [pid 27007] 03:52:48 mprotect(0x7f9766226000, 4096, PROT_NONE) = 0 <3.631704>

But I didn't identify the reaszon.

Update Aug 14, 2018 I found it is a JVM STW event. I debugged JVM with options below

 -XX:+PrintGCApplicationStoppedTime
 -XX:+PrintSafepointStatistics
 -XX:PrintSafepointStatisticsCount=1
 -XX:+SafepointTimeout
 -XX:SafepointTimeoutDelay=500

Got some log below

              vmop                    [threads: total initially_running wait_to_block]    [time: spin block sync cleanup vmop] page_trap_count
        488.188: no vm operation      [      73          1              1    ]      [     1     0  3301     0     0    ]  1

        2018-08-13T22:16:09.744-0400: 491.491: Total time for which application threads were stopped: 3.3021375 seconds, Stopping threads took: 3.3018193 seconds

The strange thing is that spin/block time is ZERO while sync time is 3301. I compiled a JVM based on open jdk 1.8 and add some debug log to it, I found it was blocked on the codes below,

     void SafepointSynchronize::begin() {
     ... ...

       if (UseCompilerSafepoints && DeferPollingPageLoopCount < 0) {
         // Make polling safepoint aware
         guarantee (PageArmed == 0, "invariant") ;
         PageArmed = 1 ;
         os::make_polling_page_unreadable();
       }
       ... ...
     }  

In function os::make_polling_page_unreadable, calling ::mprotect which has semphore dependency,

down_write(&current->mm->mmap_sem);

I doubt semophore mmap_sem contention leads to this STW event. But I don't know which function leads to this? Any help here?

---

Original Question

I am now testing Kafka's performance. I created a topic in a cluster of 6 nodes with 36 partitions and 4 replicas. One zookeeper node runs on a seperate node.

kafka-topics --create --topic kf.p36.r4 --zookeeper l2 --partitions 36 --replication-factor 4

[root@g9csf002-0-0-3 kafka]#  kafka-topics --describe --zookeeper l2 --topic kf.p36.r4
Topic:kf.p36.r4 PartitionCount:36   ReplicationFactor:4 Configs:
Topic: kf.p36.r4    Partition: 0    Leader: 1   Replicas: 1,5,6,2   Isr: 5,2,6,1
Topic: kf.p36.r4    Partition: 1    Leader: 2   Replicas: 2,6,1,3   Isr: 1,3,6,2
Topic: kf.p36.r4    Partition: 2    Leader: 3   Replicas: 3,1,2,4   Isr: 3,4,2,1
Topic: kf.p36.r4    Partition: 3    Leader: 4   Replicas: 4,2,3,5   Isr: 3,2,4,5
Topic: kf.p36.r4    Partition: 4    Leader: 5   Replicas: 5,3,4,6   Isr: 3,6,4,5
Topic: kf.p36.r4    Partition: 5    Leader: 6   Replicas: 6,4,5,1   Isr: 4,5,6,1
Topic: kf.p36.r4    Partition: 6    Leader: 1   Replicas: 1,6,2,3   Isr: 3,6,2,1
Topic: kf.p36.r4    Partition: 7    Leader: 2   Replicas: 2,1,3,4   Isr: 3,4,2,1
Topic: kf.p36.r4    Partition: 8    Leader: 3   Replicas: 3,2,4,5   Isr: 3,2,4,5
Topic: kf.p36.r4    Partition: 9    Leader: 4   Replicas: 4,3,5,6   Isr: 3,6,4,5
Topic: kf.p36.r4    Partition: 10   Leader: 5   Replicas: 5,4,6,1   Isr: 4,5,6,1
Topic: kf.p36.r4    Partition: 11   Leader: 6   Replicas: 6,5,1,2   Isr: 5,2,6,1
Topic: kf.p36.r4    Partition: 12   Leader: 1   Replicas: 1,2,3,4   Isr: 3,4,2,1
Topic: kf.p36.r4    Partition: 13   Leader: 2   Replicas: 2,3,4,5   Isr: 3,2,4,5
Topic: kf.p36.r4    Partition: 14   Leader: 3   Replicas: 3,4,5,6   Isr: 3,6,4,5
Topic: kf.p36.r4    Partition: 15   Leader: 4   Replicas: 4,5,6,1   Isr: 4,5,6,1
Topic: kf.p36.r4    Partition: 16   Leader: 5   Replicas: 5,6,1,2   Isr: 5,2,6,1
Topic: kf.p36.r4    Partition: 17   Leader: 6   Replicas: 6,1,2,3   Isr: 3,2,6,1
Topic: kf.p36.r4    Partition: 18   Leader: 1   Replicas: 1,3,4,5   Isr: 3,4,5,1
Topic: kf.p36.r4    Partition: 19   Leader: 2   Replicas: 2,4,5,6   Isr: 6,2,4,5
Topic: kf.p36.r4    Partition: 20   Leader: 3   Replicas: 3,5,6,1   Isr: 3,5,6,1
Topic: kf.p36.r4    Partition: 21   Leader: 4   Replicas: 4,6,1,2   Isr: 4,2,6,1
Topic: kf.p36.r4    Partition: 22   Leader: 5   Replicas: 5,1,2,3   Isr: 3,5,2,1
Topic: kf.p36.r4    Partition: 23   Leader: 6   Replicas: 6,2,3,4   Isr: 3,6,2,4
Topic: kf.p36.r4    Partition: 24   Leader: 1   Replicas: 1,4,5,6   Isr: 4,5,6,1
Topic: kf.p36.r4    Partition: 25   Leader: 2   Replicas: 2,5,6,1   Isr: 1,6,2,5
Topic: kf.p36.r4    Partition: 26   Leader: 3   Replicas: 3,6,1,2   Isr: 3,2,6,1
Topic: kf.p36.r4    Partition: 27   Leader: 4   Replicas: 4,1,2,3   Isr: 3,4,2,1
Topic: kf.p36.r4    Partition: 28   Leader: 5   Replicas: 5,2,3,4   Isr: 3,2,4,5
Topic: kf.p36.r4    Partition: 29   Leader: 6   Replicas: 6,3,4,5   Isr: 3,6,4,5
Topic: kf.p36.r4    Partition: 30   Leader: 1   Replicas: 1,5,6,2   Isr: 5,2,6,1
Topic: kf.p36.r4    Partition: 31   Leader: 2   Replicas: 2,6,1,3   Isr: 1,3,6,2
Topic: kf.p36.r4    Partition: 32   Leader: 3   Replicas: 3,1,2,4   Isr: 3,4,2,1
Topic: kf.p36.r4    Partition: 33   Leader: 4   Replicas: 4,2,3,5   Isr: 3,2,4,5
Topic: kf.p36.r4    Partition: 34   Leader: 5   Replicas: 5,3,4,6   Isr: 3,6,4,5
Topic: kf.p36.r4    Partition: 35   Leader: 6   Replicas: 6,4,5,1   Isr: 4,5,6,1   

I run two instances of producers, kafka-producer-perf-test

kafka-producer-perf-test --topic kf.p36.r4  --num-records  600000000 --record-size 1024 --throughput 120000 --producer-props bootstrap.servers=b3:9092,b4:9092,b5:9092,b6:9092,b7:9092,b8:9092 acks=1 

The total taffic is 240k tps and every message is 1024 bytes. When I run 240k tps traffic, everything is OK at first, but after some time, some error information appeared.

[root@g9csf002-0-0-1 ~]# kafka-producer-perf-test --topic kf.p36.r4  --num-records  600000000 --record-size 1024 --throughput 120000 --producer-props bootstrap.servers=b3:9092,b4:9092,b5:9092,b6:9092,b7:9092,b8:9092 acks=1 
599506 records sent, 119901.2 records/sec (117.09 MB/sec), 4.8 ms avg latency, 147.0 max latency.
600264 records sent, 120052.8 records/sec (117.24 MB/sec), 2.0 ms avg latency, 13.0 max latency.
599584 records sent, 119916.8 records/sec (117.11 MB/sec), 1.9 ms avg latency, 13.0 max latency.
600760 records sent, 120152.0 records/sec (117.34 MB/sec), 1.9 ms avg latency, 13.0 max latency.
599764 records sent, 119904.8 records/sec (117.09 MB/sec), 2.0 ms avg latency, 35.0 max latency.
276603 records sent, 21408.9 records/sec (20.91 MB/sec), 103.0 ms avg latency, 10743.0 max latency.
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.

I studied the logs of kafka broker, and found there are something wrong with the communication between the brokers and zookeeper.

  [2018-08-06 01:28:02,562] WARN Client session timed out, have not heard from server in 7768ms for sessionid 0x164f8ea86020062 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,562] INFO Client session timed out, have not heard from server in 7768ms for sessionid 0x164f8ea86020062, clo

the zookeeper client is zookeeper-3.4.10.jar, I download the codes and add some logs to src/java/main/org/apache/zookeeper/ClientCnxn.java

and found SendThread may be blocked sometimes when access the variable state

  [2018-08-06 01:27:54,793] INFO ROVER: start of loop. (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:27:54,793] INFO ROVER: state = CONNECTED (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:27:54,793] INFO ROVER: to = 4000 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:27:54,793] INFO ROVER: timeToNextPing = 2000 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:27:54,793] INFO ROVER: before clientCnxnSocket.doTransport, to = 2000 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:27:56,795] INFO ROVER: after clientCnxnSocket.doTransport (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: state = CONNECTED (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: start of loop. (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: state = CONNECTED (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: to = 1998 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: timeToNextPing = -1002 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: sendPing has done. (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: before clientCnxnSocket.doTransport, to = 1998 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: after clientCnxnSocket.doTransport (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: state = CONNECTED (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: start of loop. (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: state = CONNECTED (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,561] INFO ROVER: to = -3768 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,562] WARN Client session timed out, have not heard from server in 7768ms for sessionid 0x164f8ea86020062 (org.apache.zookeeper.ClientCnxn)
  [2018-08-06 01:28:02,562] INFO Client session timed out, have not heard from server in 7768ms for sessionid 0x164f8ea86020062, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

You can found between 2018-08-06 01:27:56 and 2018-08-06 01:28:02 , the thread is blocked, do nothing. the changed codes is shown below,

                // If we are in read-only mode, seek for read/write server
                if (state == States.CONNECTEDREADONLY) {
                    long now = System.currentTimeMillis();
                    int idlePingRwServer = (int) (now - lastPingRwServer);
                    if (idlePingRwServer >= pingRwTimeout) {
                        lastPingRwServer = now;
                        idlePingRwServer = 0;
                        pingRwTimeout =
                            Math.min(2*pingRwTimeout, maxPingRwTimeout);
                        pingRwServer();
                    }
                    to = Math.min(to, pingRwTimeout - idlePingRwServer);
                }

                LOG.info("ROVER: before clientCnxnSocket.doTransport, to = " + to );
                clientCnxnSocket.doTransport(to, pendingQueue, outgoingQueue, ClientCnxn.this);
                LOG.info("ROVER: after clientCnxnSocket.doTransport");
                LOG.info("ROVER: state = " + state);
            } catch (Throwable e) {

the kafka installed is confluent-kafka-2.11, and java is

[root@g9csf0002-0-0-12 kafka]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) 

Now I don't know how to fix the problem, could anyone shed some lights on this?

Answer 1

I ran into this problem before, sometimes the Kafka JVM would garbage collect for a long time, or there was something weird going on network wise internally. I noticed the timeouts were all around the 6 second or 7 second mark in our case (which seems similar to yours). The thing is, Kafka freaks out if it can't to zookeeper within the alloted time period, and it starts reporting underreplicated partitions, bringing down the whole cluster every so often. So we increased the timeout to 15 seconds, if I recall correctly, and everything worked just fine after that, with zero errors.

These are the coresponding settings on the kafka brokers:

zookeeper.session.timeout.ms    Default: 6000ms
zookeeper.connection.timeout.ms  

We changed both, IIRC, but you should try changing the session config first.