全程控制在Apache Kafka

Question

I want to run tests to measure the latency and throughout of different frameworks. I send messages to a kafka topic using an script that reads from a txt file. This frameworks consumes events from this input topic and produces into an output topic. I have two questions about this:

A) Lets say I want to send 400 events per second. How do I control that? Its something that is controlled by the script that sends the data or it can be configured in Kafka?

B) If I can control throughput tuning Kafka parameters, how do I gradually increment the amount of events sent (dinamically)?

thank you very much!

Answer 1

Throughput can be controlled in Kafka by enforcing Quotas on clients. But the catch is it's enforced in terms of Bytes/sec and not number of messages per second.

Since it's for testing purposes you can define the quota like this in the Kafka config file (server.properties):

quota.producer.default=100
quota.consumer.default=100

Do note, that this will mean it will apply this throttling on all the topics. Also, 100 here means 100 bytes.

If you want to enforce quota on specific producer or consumer clients you can do that using:

quota.producer.override="clientA:4M,clientB:6M"

Which means regardless of whatever the default Quota is, producer with client.id "ClientA" can produce at a max of 4Mbps.

As, for the dynamic part, instead of having to set these on the property file manually, you can use the kafka-config.sh file to set these configs:

bin/kafka-configs.sh  --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024' --entity-name clientA --entity-type clients

Which means, I want to enforce a throttle on a client with clientID "clientA" and the throttling is on produce-limit that needs to be set at max of 1024 bytes.

You can programmatically use this script to increase of decrease quotas dynamically.