Why Kafka cannot use Java Serializable interface

Question

In plain old Java, if I implement a class with Serializable interface, Java takes care of serializing an instance of the class and vice versa. Why do we need to supply a separate Serializer and Deserializer class if we use Kafka? Why can't Kafka use the same mechanism as Java Runtime Engine for serializing and deserializing an instance if the class implements Serializable? Can you please explain this?

Answer 1

To answer your questions, Kafka utilizes a separate Serializer and Deserializer class so that applications can choose which format of serialization to use. It is a pluggable method of adding flexibility.

Kafka does "come with" two default serializers, a byte and string deserializer, as you note neither of these utilizes Java serialization. As other comments have mentioned java serialization is often seen when the writer and reader applications are tightly coupled, use the same versions of java etc. it's issues become more apparent in environments where these assumptions can't be made. Environments like those Kafka is used in. There is also the matter of Kafka supporting other programming languages in its API's.

If this was a hard requirement you could use the default byte deserializer to read the messages from java as bytes, and post process these. The messages you'd write would be a byte string corresponding to your java serialized object. This basically uses Kafka as just a byte pass through.