logstash 从两个索引上的 filebeat 写入日志

Question

I installed the elk stack on a server, and on another server I installed filebeat to send syslog on filebeats-[data] indexes and it works fine. Now, on the elk server I configured another input in logstash to send a json file on json_data indexes and it work fine but now I find the filebeat log on both indexes and I don't understand why. I want the filebeat log only on filebeat-[data] index and not on json_data index. Where do I wrong?

This is my logstash conf file

input {
  file {
    path => "/home/centos/json/test.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  json {
    source => "message"
  }
}
output {
  elasticsearch {
    hosts => "http://10.xxx.xxx.xxx:9200"
    index => "json_data"
  }
}

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
  elasticsearch {
    hosts => "http://10.xxx.xxx.xxx:9200"
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
}

I tried different configuration, I tried also to delete the json.conf and in this case filebeat write only on the filebeat-[data] index

Answer 1

For the logs coming from filebeat to logstash, you can set the index name in filebeat configuration. In this case, logstash will not populate or manipulate the index name, ofcourse you need to remove the index part from logstash's filebeat config as well.

For json_file, keep the config as is, no need to change anything there.

To set custom index name in filebeat, you can refer: https://www.elastic.co/guide/en/beats/filebeat/current/change-index-name.html