Avro, Kafka message keys inserting extraneous characters

Question

I have my topic built like this

producer = AvroProducer(
            config.confluent.connection_config(config.confluent.CONNECTION_MODE_PRODUCER),
            default_value_schema=get_avro_schema_object(),
            default_key_schema=avro.loads('{"type": "string"}'))

producer.produce(topic=topic, value=d, key=str(d['identifier']))

that all seems fine, but when i publish, i get keys like this

print 'build_pipe_dev';
Format:AVRO
12/9/19, 12:04:23 PM EST, +$something-unique-1, {"identifier": "something-unique-1", "event_type": "build_queued", "build_request": null, "build_queued": {"jobid": "job.torque.yup"}, "build_started": null, "build_finished": null, "static_analyze_request": null}

See, in my message key field, instead of it being something-unique-1 , its +$something-unique-1 . I scratched my head all morning about this and i realized that its avro inserting its mojo into the field for the serializer/deserializer. problem is, when i don't include a key schema, the confluent_kafka library throws an error because of this in the produce function

if key is not None:
            if key_schema:
                key = self._serializer.encode_record_with_schema(topic, key_schema, key, True)
            else:
                raise KeySerializerError("Avro schema required for key")

I have seen in other posts where people monkey patched the confluent_kafka library, but is there the "proper" way to solve this?

i suppose, theres another thing to consider, is it simply working properly and I'm overreacting?

Answer 1

KSQL only supports STRING keys currently. It won't deserialise your Avro properly, hence the strange characters you're seeing in the key field.

If you use kafkacat, for example, you'll be able to consume your message and view the Avro key correctly.