I would like to know how to set BATCH SIZE (example I want a batch to have 10000 records) in TRIDENT. I have the below configuration in my code and this fetches approximately 250 records per batch from Kafka. Can I increase this to 10000*1024? or is there a way to set No of records per batch.
TridentKafkaConfig tridentKafkaConfig = new TridentKafkaConfig(hosts, topic, consumerGroupId);
tridentKafkaConfig.scheme = new SchemeAsMultiScheme(new XScheme());
tridentKafkaConfig.fetchSizeBytes = 1024*1024;
You can increase the batch size by changing " tridentKafkaConfig.fetchSizeBytes " property. Also, batch size is related with number of brokers and number of partitions. For example, if you have 2 brokers and 3 partitions for each broker that means the total count of partition is 6. By this way, the batch size equals to tridentKafkaConfig.fetchSizeBytes X total partition count . if we assume that the tridentKafkaConfig.fetchSizeBytes is 1024X1024, the batch size equals to 6 MB.(3x2x1024x1024)bytes
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.