Not able to consume large messages with Kafka (over 1.3 MB)?

Question

I followed the link How can I send large messages with Kafka (over 15MB)? to resolve the kafka msg limit issue. But no luck

I tried increasing

A.) On Broker:

message.max.bytes=15728640 
replica.fetch.max.bytes=15728640

B.) On Consumer: fetch.message.max.bytes=15728640

Still facing the same problem. Not able to consume data that is over 1.3 MB on a particular topic

In my application, a msg in sent on a topic from python code and is consumed on node server.

Answer 1

Kafka does have strict restriction over the size of data; default is 1MB . I believe you have missed topic level config.

There are multiple configs at different levels:

You have a broker setting message.max.bytes (default is 1000012) http://kafka.apache.org/documentation/#brokerconfigs

There is a topic level config max.message.bytes (default is 1000012) http://kafka.apache.org/documentation/#topicconfigs

Producer has max.request.size (default is 1048576) http://kafka.apache.org/documentation/#producerconfigs

Consumer has max.partition.fetch.bytes (default is 1048576) http://kafka.apache.org/documentation/#consumerconfigs