增加kafka的吞吐量

Question

I am reading data from a CSV file and pushing that into 3 Kafka brokers using Java code.

But, I am getting only 5 message/sec as throughput, while my expectation is about 200 message/sec

I am using asynchronous mode for pushing the data

below is my producers.properties

buffer.memory: 100000000 
batch.size: "3276800"
linger.ms: "5" 
compression.type: lz4 
retries: default 
send.buffer.bytes: 10458760  
connections.max.idle.ms: 10000

Any help would be deeply appreciated.

thanks.

Answer 1

Why write code to do this? Just use Kafka Connect (which is part of Apache Kafka) and https://github.com/jcustenborder/kafka-connect-spooldir which includes support for CSV files. Chances are this is going to perform better, and it's easier to maintain, to deploy, etc.