KafkaProducer not producing message > 1 MB to topic

Question

I am using kafka_2.11-2.2.1. I have below settings in server.properties.

message.max.bytes=20971520
replica.fetch.max.bytes=23068672

Producer configuration at code level.

@Bean("serviceKafkaTemplate")
public KafkaTemplate kafkaTemplate(@Autowired final ProducerFactory producerFactory) {
    return new KafkaTemplate(producerFactory);
}

 @Bean
public ProducerFactory<String, Object> producerFactory() {
    return new DefaultKafkaProducerFactory(props());
}

 private Properties props() {
    Properties properties = new Properties();     
    properties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");
    properties.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 10000);
    properties.put(ProducerConfig.BATCH_SIZE_CONFIG, 1);
    properties.put(ProducerConfig.LINGER_MS_CONFIG, 1);
    properties.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);
    properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    return properties;
}

@Autowired
private KafkaTemplate serviceKafkaTemplate;

serviceKafkaTemplate.send("topic_name", data);

I am only talking about the producer not the consumer. When I check the offset via cmd, it doesn't increase which means producer is not sending the data. I data size is less than 1 MB then it is working and current size is 1.75 MB. Also there was nothing related to this in kafka server.log file. Please let me know if I am miss any configuration.

Answer 1

Please take a look on the property

max.request.size=1048576

Description out of the current documentation:

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

Think this should help you. Please also take a look into the documentation.

Apache Kafka documentation - producer configs