Datastream generator for Apache Kafka

Question

I'm searching for a simple event- / datastream generator for the Kafka-broker to run some performance tests on the streaming tools of the Hadoop framework. Found nothing suitable so far. It should be able to send a lot of (mostly equal) messages in a very short periot of time (milliseconds).

Thanks!

Answer 1

You can use kafka-producer-perf-test.sh tool that is shipped with Kafka 0.9+. It allows you to produce a number of messages of a given size. This tool is just a producer that allows you send a batch of messages and collect statistics.

In our performance tests we use this tool (or our own performance producer but that is similar to this) and when we want to increase the load we run several instances in parallel into different hosts.