Trying to create a generic Kafka consumer factory

Question

I'm trying to implement a generic Kafka consumer factory. By generic I mean that the factory will receive generics K,V that will indicate the types of the key and the value of the consumed message.

public class KafkaConsumerFactory<K, V> {
    private Properties properties;

In my constructor I'm calling a method that responsible for validating that the generics K,V are compatible with the deseralization classes:

    public KafkaConsumerFactory(Properties p) {
       ....
       ....
       validateGenericsWithDeserializationProperties();
}

The relevant function:

private void validateGenericsWithDeserializationProperties() throws Exception {
    try {
        String keyDeserializerClass = (String) (this.properties.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG));
        String valueDeserializerClass = (String) (this.properties.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG));
        Deserializer<K> keyDeserializer = (Deserializer<K>) Class.forName(keyDeserializerClass).getConstructor().newInstance();
        Deserializer<V> valueDeserializer = (Deserializer<V>) Class.forName(valueDeserializerClass ).getConstructor().newInstance();
        System.out.println(keyDeserializer.toString());
        System.out.println(valueDeserializer.toString());
    } catch (Exception e) {
        throw new Exception("There is an incompatibility with the deserialization parameters in the properties with the generics <K,V>");
    }

}

But it seems that when I pass generics K,V that aren't compatible with the deserializer class the code doesn't throw exception, for example:

K=String,
V=Long,
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

I was expecting an exception on the downcasting on the following lines:

    Deserializer<K> keyDeserializer = (Deserializer<K>)(Class.forName(keyDeserializerClass).getConstructor().newInstance());
    Deserializer<V> valueDeserializer = (Deserializer<V>)Class.forName(valueDeserializerClass).getConstructor().newInstance();

I understand that those lines won't throw exception because of erasure of generic types.

Is there any other way to check for incompatibility between the deserialization class to the type of generic before trying to consume data?

Answer 1

    Deserializer<K> keyDeserializer = (Deserializer<K>) Class.forName(keyDeserializerClass).getConstructor().newInstance();
    Deserializer<V> valueDeserializer = (Deserializer<V>) Class.forName(valueDeserializerClass ).getConstructor().newInstance();

Why do you want to explicity create Deserializer object?

1. It will automatically be created as per the one specified in the properties.

2. If you want to verify (make sure) the appropriate deserializer is put for the appropriate key/value type, you can take the keyClass, valueClass in the constructor of your KafkaConsumerFactory

  public KafkaConsumerFactory(Properties p, Class<K> keyClass, Class<V> valueClass) {
     p.putIfAbsent("key.deserializer", getDeserializer(keyClass).getName());
     p.putIfAbsent("value.deserializer", getDeserializer(valueClass).getName());
    }

and then you can automatically map the keyClass , valueClass to appropriate deserializer and override them in the properties.

Class<Deserializer> getDeserializer(Class clazz) {
   if(Long.class.equals(clazz)) { return LongDeserializer.class; }
   else if (...)
   ...
}

If you want to support new custom deserializers, you can use an interface like CustomDeserializer for example that has a method called canBeProcessed(Class clazz) which determines if the object can be deserialized and ensure that this interface is implemented by all your custom deserializers and that the method is overriden.

If you are looking to support already existing deserializers, you can write the code to check the compatability of the deserializer with a certain class in your KafkaConsumerFactory itself.

Usually, serializers/deserializers are mapped to certain class(es), which means that you don't need an object to verify if it can be serialized/deserialized because the data in the object has pretty less (if not nothing) to do with the actual serialization/deserialization.

In case, if the class type information isn't enough for finding out whether it's object can be deserialized or not, the option left is to create a dummy object and then pass it to the deserialize() method. If it raises an exception, then it is invalid, not otherwise.