加载Avro时，Kafka Connect S3接收器引发IllegalArgumentException

Question

I'm using qubole's S3 sink to load Avro data into S3 in Parquet format.

In my Java application I create a producer

Properties props = new Properties();
props.put("bootstrap.servers", KafkaHelper.getServers());
props.put("key.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
return new KafkaProducer<byte[], byte[]>(props);

Then convert a GenericRecord into byte[] format:

GenericRecord avroRecord = new GenericData.Record(avroSchema);
Injection<GenericRecord, byte[]> recordInjection = GenericAvroCodecs.toBinary(avroSchema);

for (Map.Entry<String, ?> entry : map.entrySet()) {
    String key = entry.getKey();
    Object value = entry.getValue();
    avroRecord.put(key, value);
}

ProducerRecord<byte[], byte[]> record = new ProducerRecord<>(topic, recordInjection.apply(avroRecord));
producer.send(record);

I use the following values in my Kafka Connect properties:

key.converter=com.qubole.streamx.ByteArrayConverter
value.converter=com.qubole.streamx.ByteArrayConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

And the following configuration options in my file sink properties:

connector.class=com.qubole.streamx.s3.S3SinkConnector
format.class=io.confluent.connect.hdfs.parquet.ParquetFormat

When I run the connector I get the following error message: 'java.lang.IllegalArgumentException: Avro schema must be a record'.

I'm pretty new to Kafka Connect and I know that a Schema Registry server can be set up--but I don't understand whether or not the sink needs the registry to convert the Avro data to Parquet or if this is some kind of formatting or configuration problem on my end. What kind of data format does a "record" refer to in the context of this error? Any direction or help would be much appreciated.

Answer 1

The ByteArrayConverter is not going to do any translation of data: instead of actually doing any serialization/deserialization, it assumes the connector knows how to handle raw byte[] data. However, the ParquetFormat (and in fact most formats) cannot handle just raw data. Instead, they expect data to be deserialized and structured as a record (which you can think of as a C struct, a POJO, etc).

Note that the qubole streamx README notes that ByteArrayConverter is useful in cases where you can safely copy the data directly. Examples would be if you have the data as JSON or CSV. These don't need deserialization because the bytes for each Kafka record's value can simply be copied into the output file. This is a nice optimization in those cases, but not generally applicable to all output file formats.