Testing Kafka producer throughput

Question

We have a Kafka cluster consists of 3 nodes each with 32GB of RAM and 6 core 2.5 CPU.

We wrote a kafka producer that receive tweets from twitter and send it to Kafka in batches of 5000 tweets.

In the Producer we uses producer.send(list<KeyedMessages>) method.

The avg size of the tweet is 7KB.

Printing the time in milliseconds before and after the send statement to measure the time taken to send 5000 messages we found that it takes about 3.5 seconds.

Questions

Is the way we test the Kafka performance correct?

Is using the send method that takes list of keyed messages the correct way to send batch of messages to Kafka? Is there any other way?

What are the important configurations that affects the producer performance?

Answer 1

You're measuring only the producer side? That metric tells you only how much data you can store in a unit of time.

Maybe that's what you wanted to measure, but since the title of your question is "Kafka performance", I would think that you'd actually want to measure the throughput, ie how long does it take for a message to go though Kafka (usually referred to as end-to-end latency).

You'd achieve that by measuring the difference in time between sending a message and receiving that message on the other side, by a consumer.

If the cluster is configured correctly (default configuration will do), you should see latency ranging from only a couple of ms (less than 10ms), up to 50ms (few tens of milliseconds).

Kafka is able to do that because messages you read by the consumer don't even touch the disk, cuz' they are still in RAM (page cache and socket buffer cache). Keep in mind that this only works while you're able to "catch up" with your consumers, ie don't have a large consumer lag. If a consumer lags behind producers, the messages will eventually be purged from cache (depending on the rate of messages - how long it takes for the cache to fill up with new messages), and will thus have to be read from disk. Even that is not the end of the world (order of magnitude slower, in the range of low 100s of ms), because messages are written consecutively, one by one is a straight line, which is a single disk seek.

BTW you'd want to give Kafka only a small percentage of those 32GB, eg 5 to 8GB (even G1 garbage collector slows down with bigger sizes) and leave everything else unassigned so OS can use it for page and buffer cache.