KafkaError与Confluent Python Kafka使用者不支持的压缩编解码器0x3

Question

I set up Confluent Python Kafka consumer on my laptop (Ubuntu 17) and everything is fine, I can listen to remote topics and receive messages.

But when I try to set it up on a server (Ubuntu 16), there is what seems to be a compression issue. The data comes from Divolte and is compressed with LZ4.

The data is received successfully without any error when first connecting to a topic, but after closing and re-opening the consumer, a first message is received and an error is thrown:

<cimpl.Message object at 0x7f089db67180>
KafkaError{code=_NOT_IMPLEMENTED,val=-170,str="Unsupported compression codec 0x3"}

I think it's not from Divolte data source but more like a message from Kafka, but I cannot read its value since the error happens before (msg is printed and then we jump to elif for the error):

c = Consumer({'bootstrap.servers': server['server'], 'group.id': 'mygroup',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
c.subscribe([server['topic']])
running = True
while running:
    msg = c.poll()
    print(msg)
    if not msg.error():
        msg_value = msg.value()
        print(msg_value)
    elif msg.error().code() != KafkaError._PARTITION_EOF:
        print(msg.error())
        running = False

Answer 1

Sounds like this problem which should be fixed in 0.10.0 or higher

https://issues.apache.org/jira/browse/KAFKA-3160

Answer 2

确实，这似乎是一个版本问题，但显然没有必要使用0.10，我按照http://docs.confluent.io/current/installation.html重新安装存储库和融合平台，然后从0.9.1更新了librdkafka1和librdkafka-dev。到0.9.5，它解决了我的问题！