ElasticSearch Logstash 未连接“连接被拒绝”- Docker

Question

我需要帮助，（谁会想到？对吧？哈哈）

几天后我有一个工作面试，这对我来说意味着世界为它做好充分准备并有一些工作示例。

我正在尝试通过 logstash、elasticsearch 从 kafka 建立一个 stream 数据的 ELK 管道，最后从 Kibana 读取它。 通常。 我正在使用容器，但是二重奏 logstash - elasticsearch 让我患上了动脉瘤。

其他一切都很好。 我检查了 kafka 的日志，一切正常。 Kibana 也收集到 elasticsearch 就好了。 但是logstash和es实在不想搭配。

这是设置

docker-compose.yml

version: '3.6'
services:
  elasticsearch:
    image: elasticsearch:8.6.0
    container_name: elasticsearch
    #restart: always
    volumes:
    - elastic_data:/usr/share/elasticsearch/data/
    environment:
      cluster.name: elf-kafka-cluster
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      discovery.type: single-node    
      xpack.security.enabled: false
    ports:
    - '9200:9200'
    - '9300:9300'
    networks:
      - elk

  kibana:
    image: kibana:8.6.0
    container_name: kibana
    #restart: always       
    ports:
    - '5601:5601'
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200  
    depends_on:
      - elasticsearch  
    networks:
      - elk

  logstash:
    image: logstash:8.6.0
    container_name: logstash
    #restart: always
    volumes:
    - type: bind
      source: ./logstash_pipeline/
      target: /usr/share/logstash/pipeline
      read_only: true
    command: logstash -f /home/ettore/Documenti/Portfolio/ELK/logstash/logstash.conf 
    depends_on:
      - elasticsearch
    ports:
    - '9600:9600'
    environment:
      xpack.monitoring.enabled: true
    #   LS_JAVA_OPTS: "-Xmx256m -Xms256m"    
    links:
      - elasticsearch
    networks:
      - elk

volumes:
  elastic_data: {}

networks:
  elk:
    driver: bridge

logstash 配置文件

input {
    kafka {
        bootstrap_servers => "localhost:9092"
        topics => ["topic"]
    }
}

output {
    elasitcsearch {
        hosts => ["http://localhost:9200"]
        index => "topic"
        workers => 1
    }
}

这些是我编写时的 logstash 错误日志：

logstash       | [2023-01-17T13:59:02,680][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash       | Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
logstash       | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash       | [2023-01-17T13:59:04,711][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash       | [2023-01-17T13:59:05,373][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused>}
logstash       | [2023-01-17T13:59:05,379][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused"}
logstash       | [2023-01-17T13:59:05,436][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused>}
logstash       | [2023-01-17T13:59:05,444][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash       | [2023-01-17T13:59:05,449][WARN ][logstash.licensechecker.licensereader] Attempt to validate Elasticsearch license failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused"}
logstash       | [2023-01-17T13:59:05,477][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
logstash       | [2023-01-17T13:59:05,567][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash       | [2023-01-17T13:59:05,661][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/home/ettore/Documenti/Portfolio/ELK/logstash/logstash.conf"}
logstash       | [2023-01-17T13:59:05,664][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
logstash       | [2023-01-17T13:59:06,333][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash       | [2023-01-17T13:59:06,411][INFO ][logstash.runner          ] Logstash shut down.
logstash       | [2023-01-17T13:59:06,419][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
logstash       | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash       |    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
logstash       |    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
logstash       |    at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

这是为了证明一切都按 es 的预期工作（或者看起来如此）

netstat -an | grep 9200

tcp        0      0 0.0.0.0:9200            0.0.0.0:*               LISTEN     
tcp6       0      0 :::9200                 :::*                    LISTEN     
unix  3      [ ]         STREAM     CONNECTED     49200  

我已经查看了所有内容，这 100% 不是重复的，因为我已经尝试了所有内容。 我真的想不通。 希望任何人都可以提供帮助。

谢谢你的时间。

Answer 1

你应该设置 logstash.yml

1. 使用以下值创建 logstash.yml：

http.host：“0.0.0.0”

xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]

2. 在您的 docker-compose.yml 中，在 Logstash 容器中添加另一个卷，如下所示：

./logstash.yml:/usr/share/logstash/config/logstash.yml

此外，它很适合在重启条件下运行。