在Confluent Python Avro Producer中指定压缩类型

Question

Is there a way to specify the compression.type in producer configs while using the AvroProducer in Confluent's Kafka (python)?

I tried the following:

from confluent_kafka import avro
from confluent_kafka.avro import AvroProducer

from myconfigs import BOOTSTRAP_SERVER, SCHEMA_REGISTRY_URL, KEY_SCHEMA, VALUE_SCHEMA

avroProducer = AvroProducer({'bootstrap.servers': BOOTSTRAP_SERVER, 'schema.registry.url': SCHEMA_REGISTRY_URL, 'compression.type': 'gzip'},
                            default_key_schema=KEY_SCHEMA, default_value_schema=VALUE_SCHEMA)

Got the following error when running this:

Traceback (most recent call last):
  File "confluent_click.py", line 47, in <module>
default_key_schema=KEY_SCHEMA, default_value_schema=VALUE_SCHEMA)
  File "/usr/local/lib/python3.6/site-packages/confluent_kafka/avro/__init__.py", line 38, in __init__
    super(AvroProducer, self).__init__(config)
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="No such configuration property: "compression.type""}

Also tried specifying compression_type = 'gzip' as a param to AvroProducer() as

avroProducer = AvroProducer({'bootstrap.servers': BOOTSTRAP_SERVER, 'schema.registry.url': SCHEMA_REGISTRY_URL},
                            default_key_schema=KEY_SCHEMA, default_value_schema=VALUE_SCHEMA, compression_type='gzip')

I didn't expect this to succeed and it didn't.

Traceback (most recent call last):
  File "confluent_click.py", line 47, in <module>
    default_key_schema=KEY_SCHEMA, default_value_schema=VALUE_SCHEMA, compression_type='gzip')
TypeError: __init__() got an unexpected keyword argument 'compression_type'

How can I specify compression.type in the producer? I have not been able to find AvroProducer 's documentation.

Answer 1

confluent-kafka-python's configuration property for setting the compression type is called compression.codec for historical reasons (librdkafka, which predates the current Java client, based its initial configuration properties on the original Scala client which used compression.codec ).

avroProducer = AvroProducer({'bootstrap.servers': BOOTSTRAP_SERVER, 'schema.registry.url': SCHEMA_REGISTRY_URL, 'compression.codec': 'gzip'},
                            default_key_schema=KEY_SCHEMA, default_value_schema=VALUE_SCHEMA)

Note: the v0.11.4 release of confluent-kafka-python adds a compression.type alias.

Full list of configuration settings here: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md