why does kafka producer (perf test) have such low throughput / high latency?

Question

I am new to kafka and am running some performance tests. I am running a 2 machine cluster consisting of my laptop and a raspberry pi zero W (1 GHz, single-core CPU, 512 MB RAM, 802.11n wireless LAN). eventually the pi will run a single producer (java) that sends binary sensor data (ideally smaller records, eg, 10 kb, as quickly as possible) to kafka which is then read off by a consumer on my laptop or another pi. a separate application later processes the data offline. i only have 1 topic with 1 partition and no replication.

i'm currently trying to benchmark the producer performance / get an idea of it's capabilities. this is the output from the kafka-producer-perf-test.sh, showing that there is a pretty large latency and not great throughput. i've played around with the settings (like linger.ms, batch.size, acks, throughput, and record size, it seems like the larger the record size, the higher the throughput (up until ~750 kb, then decreases) but the latency either is about the same or actually decreases.

i've only been able to achieve ~0.6 MB/sec which is WAY too slow for the application i'm working on. when i decreased the record size, the throughput was even worse at ~0.10 MB/sec. since i'm new to kafka, i'm wondering is this what I should expect when using the pi zero w or is something suboptimal with my setup / producer properties, etc?

pi@raspberrypi:~/Kafka/kafka_2.12-2.4.0 $ bin/kafka-producer-perf-test.sh --topic perf_test --num-records 100 --thughput 1000000 --producer-props bootstrap.servers=192.168.0.170:9092,192.168.0.172:9092 --record-size 750000
OpenJDK Zero VM warning: G1 GC is disabled in this release.
2 records sent, 0.3 records/sec (0.25 MB/sec), 4439.5 ms avg latency, 4607.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 6909.0 ms avg latency, 9180.0 ms max latency.
5 records sent, 0.8 records/sec (0.58 MB/sec), 12783.4 ms avg latency, 15075.0 ms max latency.
5 records sent, 0.8 records/sec (0.59 MB/sec), 18587.0 ms avg latency, 21165.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 24551.2 ms avg latency, 26851.0 ms max latency.
4 records sent, 0.8 records/sec (0.55 MB/sec), 30048.0 ms avg latency, 31850.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 35323.4 ms avg latency, 37657.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 41030.4 ms avg latency, 43266.0 ms max latency.
5 records sent, 0.8 records/sec (0.59 MB/sec), 46736.6 ms avg latency, 49240.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 51716.2 ms avg latency, 53756.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 53765.8 ms avg latency, 53848.0 ms max latency.
5 records sent, 0.9 records/sec (0.61 MB/sec), 53506.2 ms avg latency, 53731.0 ms max latency.
4 records sent, 0.8 records/sec (0.55 MB/sec), 53638.3 ms avg latency, 54029.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 53690.4 ms avg latency, 54003.0 ms max latency.
5 records sent, 0.9 records/sec (0.64 MB/sec), 53371.6 ms avg latency, 53703.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 52922.0 ms avg latency, 53008.0 ms max latency.
4 records sent, 0.7 records/sec (0.53 MB/sec), 53774.3 ms avg latency, 53805.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 53788.6 ms avg latency, 53827.0 ms max latency.
5 records sent, 0.9 records/sec (0.64 MB/sec), 53317.2 ms avg latency, 53698.0 ms max latency.
5 records sent, 0.8 records/sec (0.60 MB/sec), 53373.2 ms avg latency, 53406.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 53441.8 ms avg latency, 53476.0 ms max latency.
100 records sent, 0.816993 records/sec (0.58 MB/sec), 42056.90 ms avg latency, 54029.00 ms max latency, 52990 ms 50th, 53805 ms 95th, 54029 ms 99th, 54029 ms 99.9th.

Answer 1

Messages are batched before they are sent to Kafka. More the number of messages in the batch, more can be sent in a single shot, the higher will be throughput.

If you are increasing message size, then you also have to increase batch.size so that each batch allocates more number of messages in it.

From the code you have pasted, there is no property of setting batch.size, so it will be default which is 16384 bytes (16KB) which is way less than your message size (750 kb), so that is why you see 0.3 records/sec and so on.

In batch.size documentation

No attempt will be made to batch records larger than this size.

So, you can try increasing your batch.size to allocate more messages, for example to 10Mb or so and check.