现有CDH 5.5.2集群上的Kafka配置

Question

I am installing Kafka-2.0 on my existing CDH 5.5.2 cluster, here is the procedure what i followed

1. Add services from CM
2. Selected Kafka (Before that i downloaded and distributes and activated kafka parcel on all the nodes)
3. Selected 1 nodes for KafkaBroker and 4 nodes for Kafka MirrorMaker
4. Then i updated my Destination Broker List (bootstrap.servers) property with one of the Mirror Maker node as well as Source Broker List (source.bootstrap.servers) with same node

5. Below error i am getting (log file)

    Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:331) at kafka.log.SkimpyOffsetMap.<init>(OffsetMap.scala:43) at kafka.log.LogCleaner$CleanerThread.<init>(LogCleaner.scala:186) at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83) at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) at scala.collection.immutable.Range.foreach(Range.scala:166) at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at kafka.log.LogCleaner.<init>(LogCleaner.scala:83) at kafka.log.LogManager.<init>(LogManager.scala:64) at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:601) at kafka.server.KafkaServer.startup(KafkaServer.scala:180) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37) at kafka.Kafka$.main(Kafka.scala:67) at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76) at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala) 

Answer 1

you need to increase broker_max_heap_size value to atleast 1GB and restart the kafka service from Cloudera Manager. If you still face the same issue, try to increase as per your cluster configurations

Answer 2

The stack trace shows "java.lang.OutOfMemoryError: Java heap space" - VM heap is running out of space. Increase it by setting

export KAFKA_HEAP_OPTS="-Xmx1G -Xms512M" 

in /bin/kafka-server-start.sh.