How do I set the Java options for Kafka?

Question

I've been experimenting with Kafka and saw from the documentation on the main site that you're able to set different options for the jvm like heap size and the garbage collector that it uses:

http://kafka.apache.org/documentation.html#java

What it doesn't say, however, is how/where to set these options. The application comes with a /config directory containing a lot of files used for configuration purposes but none that are for Java. It also comes with a /bin directory containing a bunch of scripts for Kafka but again, nothing really indicating how to configure Java.

So my question is, how do I configure the Java options that Kafka uses? Is it done through a file or is there a different way?

Answer 1

Modifying a script in the bin directory is highly unrecommended. When upgrading Kafka to the next version, extracting the new binaries would override the changes made in the script.

The preferred way should be to set the environment variable KAFKA_HEAP_OPTS outside the script.

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

If the var is set before starting Kafka via the script it will use the var instead of the default values defined in /bin/kafka-server-start.sh

Answer 2

Another way to do this is by modifying information written in /bin/kafka-server-start.sh :

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

or in /bin/kafka-run-class.sh :

KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"

Answer 3

Looking at kafka-run-classh.sh - kafka uses the following variables:

• $KAFKA_HEAP_OPTS
• $KAFKA_JVM_PERFORMANCE_OPTS
• $KAFKA_GC_LOG_OPTS
• $KAFKA_JMX_OPTS
• $KAFKA_LOG4J_OPTS

You can run it via:

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" 
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12346 -Dcom.sun.management.jmxremote.rmi.port=12346 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 
bin/kafka-server-start.sh -daemon config/server.properties

Answer 4

You can also set this parameters via your service definition as shown below.

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
Environment="KAFKA_HEAP_OPTS=-Xmx4G"
Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true"
Environment="JMX_PORT=9999"

[Install]
WantedBy=multi-user.target

Answer 5

To add generic JVM settings (for example user timezone) you could use KAFKA_OPTS environment variable from kafka-run-class.sh :

# Generic jvm settings you want to add
if [ -z "$KAFKA_OPTS" ]; then
  KAFKA_OPTS=""
fi

Answer 6

You can pass java parameters from command line . Eg

java -server -Xms3072m -Xmx3072m -XX:NewSize=256m -XX:MaxNewSize=256m  -classpath <long list of jars> foo.class

For a `production server config you can create a properties file or set them in Code by creating

 Properties props = new Properties();
 props.put("serializer.class", "kafka.serializer.StringEncoder");

And then supply them to producerConfig

ProducerConfig config = new ProducerConfig(props);