Logstash 管道未在 Kibana 上显示，但日志显示管道正在运行

Question

尝试设置弹性搜索、kibana 和 logstash 以从本地文件夹读取日志。 它在版本 7.xx 上运行良好，但是当我尝试升级到 8 时它却不行。Fx

我正在使用这个 YAML 文件：

version: '3.6'
services:
  Elasticsearch:
    image: elasticsearch:8.4.0
    container_name: elasticsearch
    volumes:
    - elastic_data:/usr/share/elasticsearch/data/
    environment:
    - discovery.type=single-node
    - xpack.license.self_generated.type=basic
    - xpack.security.enabled=false
    ports:
    - '9200:9200'
    - '9300:9300'
    networks:
      - elk

  Logstash:
    image: logstash:8.4.0
    container_name: logstash
    environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    - xpack.monitoring.enabled=true

    volumes:
    - ./logstash/:/logstash
    - D:/test/Logs/:/test/Logs
    command: logstash -f /logstash/logstash.conf 
    depends_on:
    - Elasticsearch
    ports:
    - '9600:9600'
    networks:
    - elk

  Kibana:
    image: kibana:8.4.0
    container_name: kibana      
    ports:
    - '5601:5601'
    environment:
    - ELASTICSEARCH_URL=http://elasticsearch:9200  
    depends_on:
    - Elasticsearch  
    networks:
    - elk
volumes:
  elastic_data: {}

networks:
  elk:

和logstash的配置：

input {
    file {
        path => "/test/Logs/test.slog"
        start_position => "beginning"
    }
}

output {
    elasticsearch {
        hosts => ["elasticsearch:9200"]
    }
}

test.slog 存在并包含日志。

logstash docker 显示以下日志：

[2022-08-27T20:40:32,592][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}
[2022-08-27T20:40:33,450][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.95}
[2022-08-27T20:40:33,451][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.94}
[2022-08-27T20:40:33,516][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-08-27T20:40:33,532][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_327fd1919fa26d08ec354604c3e1a1ce", :path=>["/test/Logs/test.slog"]}
[2022-08-27T20:40:33,559][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-08-27T20:40:33,614][INFO ][filewatch.observingtail  ][main][8992bf4e2fad9d8838262d3019319d02ab5ffdcb5b282e821574485618753ce9] START, creating Discoverer, Watch with file and sincedb collections
[2022-08-27T20:40:33,625][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}

但是当我 go 到数据 - >索引管理时什么都没有。 并且还在摄取管道中。

我究竟做错了什么？

Answer 1

在 Elasticsearch 8 中，由 logstash output 创建的索引名称遵循 pattern.ds-logs-generic-default-%{+yyyy.MM.dd} 而不是 logstash-%{+yyyy.MM.dd}

Data -> Index Management下没有this.ds索引但是可以查询到文档

您可以使用 Kibana，Management> Dev Tools 查看.ds-logs-generic索引

GET _cat/indices

要查询文档，您可以使用_search API

GET /.ds-logs-generic-default-2022.08.28-000001/_search
{
  "query": {
    "match_all": {}
  }
}

如果要指定索引名称，可以将其添加到 logstash.conf 的 output 部分，例如index => "logstash-%{+YYYY.MM.dd}"

output {
    elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
    }
}

新创建的索引将显示在 Kibana 中的Management > Data > Index Management下。 您可能需要在日志文件的末尾添加一些日志行来启动索引管道。