如何在kafka制作人中切换到自定义编码器？

Question

I am getting the following error when I try to use string as key in kafka topic.

18/05/14 17:08:26 ERROR async.DefaultEventHandler: Error serializing message for topic my_topic
java.lang.ClassCastException: java.lang.String cannot be cast to [B
    at kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:34)
    at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:130)
    at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:127)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:127)
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:53)
    at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
    at scala.collection.immutable.Stream.foreach(Stream.scala:547)
    at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
    at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)

It seems like the issue is that the default encoder

public class DefaultEncoder implements Encoder<byte[]> 

does not support string to byte

public byte[] toBytes(byte[] value) {
    return value;
}

What is the proper way of providing a custom encoder to the producer?

And do I have to change at the consumer side as well?

Answer 1

On the Producer/Consumer you can specify a different serializer rather than the default ByteArraySerializer, Here you can find the available serializers for version 1.1.0 , or you can specify your own.

In general if you are sending/receiving Strings or Json you can just use the default org.apache.kafka.common.serialization.StringSerializer and org.apache.kafka.common.serialization.StringDeserializer . For the producer/consumer the properties are:

private void configureProducer() {
    Properties props = new Properties();
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
}

private void configureConsumer() {
    Properties props = new Properties();
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringDeserializer");
}

Properties producerProps = configureProducer();
KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps);
Properties consumerProps = configureConsumer();
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps);

If you want to send your own custom bean then either convert it to json and use the default StringSerializer/Deserializer or build your own serialization/deserialization classes. You can see an example using json strings with spring boot here