ELK - Logstash + Redis - 数据复制

Question

I need a help

I have those both Logstash config files:

agent.conf

 input {
  log4j {
    type => "bdj"
    port => 25827
  }
}

filter{
    json{
        source => "message"
    }

}

output {
  stdout { 
    codec => rubydebug
  }
  redis {
    host => "127.0.0.1"
    data_type => "list"
    key => "logstash"
  }
}

The agent.conf receive the data logs by tcp e foward them to redis.

central.conf

input {
  redis {
    host => "localhost"
    type => "redis-input"
    data_type => "list"
    key => "logstash"
  }
}

filter{
  json{
    source => "message"
  } 
}

output {
  stdout { }
  elasticsearch {
    hosts => "localhost"
    index => "logstash-%{+YYYY.MM.dd}"
  }

}

The central.conf gets the redis and foward them to the elastichsarch.

My problem is that the data are been duplicated, as a loop or something like that.

I'm running logstash as a sevice on Debian;

root@logs:~# uname -a
Linux logs 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux

Take a look at the image, the same data log, at the same time, and I just sent one log request.

Any help?

Answer 1

When I find logstash doing weird things, it sometimes comes down to having two copies running or an old config file in the directory that is also being read by logstash. This isn't every problem, but it's good to check.

Answer 2

I ran into the same issue for a few days until I get what was going on : an instance of Logstash has only one pipeline processing events ! Logstash gives the ability to split the configuration into different files for readability but at the end they are all merged together into a single one configuration.

In this case, after loading central.conf and agent.conf your Logstash instance pipeline has two inputs (log4j and redis), one filter (which should be defined only once to avoid duplication) and two outputs (redis and elasticsearch). Events came from the two inputs, are processed through your filter and then pushed to the two outputs.

After an event coming from your log4j input has been pushed to the redis output, it is processed again from the redis input to the redis output. Here is your infinite loop.

The solution is to "tag" your events to selectively apply filters and outputs later on in the event's lifetime. All inputs plugins have a common option type which can be used for this. But as it is said in the documentation the type of an event is immutable, so you should not be able to change it once it is set. I would rather use the @metadata field in this context, see below.

agent.conf

input {
  log4j {
    type => "bdj"
    port => 25827
    add_field => { 
      "[@metadata][event_origin]" => "log4j"
    }
  }
}

filter {
  if [@metadata][event_origin] == "log4j" {
    json {
      source => "message"
    }
  }
}

output {
  if [@metadata][event_origin] == "log4j" {
    redis {
      host => "127.0.0.1"
      data_type => "list"
      key => "logstash"
    }
  }
}

central.conf

input {
  redis {
    host => "localhost"
    data_type => "list"
    key => "logstash"
    add_field => { 
      "[@metadata][event_origin]" => "redis"
    }
  }
}

output {
  if [@metadata][event_origin] == "redis" {
    elasticsearch {
      hosts => "localhost"
      index => "logstash-%{+YYYY.MM.dd}"
    }
  }
}