Kafka 流聚合 RecordTooLargeException

Question

Trying to aggregate data from a single stream with tumbling windows of 5 minutes. Initially it was working fine and was able to print aggregated records. Using Java 8 for development.

But later started receiving an error -

"org.apache.kafka.common.errors.RecordTooLargeException: The message is 5292482 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration"

AND now every time, after starting my app in EKS cluster, crashes within a minute, with the same error.

Tried to set following streamconfigs but they also not helped: StreamsConfig.RECEIVE_BUFFER_CONFIG (50 MB) StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG (50 MB)

Can anyone of the following will resolve this issue:

1. Using inMemoryKeyValueStore. is there any specific property to allocate some memory for inMemoryKeyValueStore?
2. OR should switch to persistentKeyValueStore?
3. Using AWS MSK so while creating cluster, define broker and topic level setting - message.max.bytes with appropriate value.

Thanks in advance.

Answer 1

It's the producer config max.request.size (as mentioned in the error message) that you need to increase to address the issue.

Note that you might need to increase the broker/topic config message.max.bytes that you mentioned in addition.