使用Kafka进行Logstash：无法解码avro

Question

我试图从Kafka队列中消耗序列化的avro事件。 使用简单的java生成器填充kafka队列。 为清楚起见，我分享了三个组成部分：

Avro架构文件

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

Java Producer代码片段 （User.class使用avro-tools生成）

    User user1 = new User();
    user1.setName("Alyssa");
    user1.setFavoriteNumber(256);
    user1.setFavoriteColor("blue");
    String topic = "MemoryTest";

    // Properties set in 'props'
    KafkaProducer<Message, byte[]> producer = new KafkaProducer<Message, byte[]>(props);

    ByteArrayOutputStream out = new ByteArrayOutputStream();
    DatumWriter<User> writer = new SpecificDatumWriter<User>(User.class);
    Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
    writer.write(user1, encoder);
    encoder.flush();
    out.close();
    byte[] serializedBytes = out.toByteArray();
    producer.send(new ProducerRecord<Message, byte[]>(topic, serializedBytes));

Logstash配置文件

input {
        kafka {
                zk_connect => "localhost:2181"
                topic_id => "MemoryTest"
                type => "standard_event"
                group_id => "butiline_dash_prod"
        reset_beginning => true
        auto_offset_reset => smallest
        codec => {
                avro => {
                    schema_uri => "/opt/ELK/logstash-1.5.4/bin/user.avsc"
                }
            }
        } 
}

output {
    stdout { 
     codec => rubydebug 
     }
}

问题

管道在logstash级别失败。 当一个新事件被推送到Kafka时，我会在logstash控制台上关注：

Alyssa�blue {:exception=>#<NoMethodError: undefined method `decode' for ["avro", {"schema_uri"=>"/opt/ELK/logstash-1.5.4/bin/user.avsc"}]:Array>, :backtrace=>["/opt/ELK/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-1.0.0/lib/logstash/inputs/kafka.rb:169:in `queue_event'", "/opt/ELK/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-1.0.0/lib/logstash/inputs/kafka.rb:139:in `run'", "/opt/ELK/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:177:in `inputworker'", "/opt/ELK/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:171:in `start_input'"], :level=>:error}

Answer 1

终于找到了错误。 而不是这个（正如Logstash网站上所建议的那样 - https://www.elastic.co/guide/en/logstash/current/plugins-codecs-avro.html ）

codec => {
    avro => {
        schema_uri => "/opt/ELK/logstash-1.5.4/bin/user.avsc"
    }
}

正确的语法是（如插件文档https://github.com/logstash-plugins/logstash-codec-avro/blob/master/DEVELOPER.md中所述 ）：

codec =>   avro {
        schema_uri => "/opt/ELK/logstash-1.5.4/bin/user.avsc"
}

我想语法已经改变了。