ElasticSearch Logstash not connecting "Connection refused" - Docker

Question

I need help, (who would have thought? right? lol)

I have a job interview in few days and it would mean the world to me to be well prepared for it and have some working examples.

I am trying to set up an ELK pipeline to stream data from kafka, through logstash, elasticsearch and finally read it from Kibana. The usual. I am making use of containers, but the duo logstash - elasticsearch are giving me an aneurism.

Everything else works perfectly fine. I've checked the logs off of kafka and that is working just fine. Kibana is collected to elasticsearch just fine as well. But logstash and es really don't want to match.

Here is the setup

docker-compose.yml

version: '3.6'
services:
  elasticsearch:
    image: elasticsearch:8.6.0
    container_name: elasticsearch
    #restart: always
    volumes:
    - elastic_data:/usr/share/elasticsearch/data/
    environment:
      cluster.name: elf-kafka-cluster
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      discovery.type: single-node    
      xpack.security.enabled: false
    ports:
    - '9200:9200'
    - '9300:9300'
    networks:
      - elk

  kibana:
    image: kibana:8.6.0
    container_name: kibana
    #restart: always       
    ports:
    - '5601:5601'
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200  
    depends_on:
      - elasticsearch  
    networks:
      - elk

  logstash:
    image: logstash:8.6.0
    container_name: logstash
    #restart: always
    volumes:
    - type: bind
      source: ./logstash_pipeline/
      target: /usr/share/logstash/pipeline
      read_only: true
    command: logstash -f /home/ettore/Documenti/Portfolio/ELK/logstash/logstash.conf 
    depends_on:
      - elasticsearch
    ports:
    - '9600:9600'
    environment:
      xpack.monitoring.enabled: true
    #   LS_JAVA_OPTS: "-Xmx256m -Xms256m"    
    links:
      - elasticsearch
    networks:
      - elk

volumes:
  elastic_data: {}

networks:
  elk:
    driver: bridge

logstash.conf

input {
    kafka {
        bootstrap_servers => "localhost:9092"
        topics => ["topic"]
    }
}

output {
    elasitcsearch {
        hosts => ["http://localhost:9200"]
        index => "topic"
        workers => 1
    }
}

These are logstash error logs when I compose up:

logstash       | [2023-01-17T13:59:02,680][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash       | Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
logstash       | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash       | [2023-01-17T13:59:04,711][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash       | [2023-01-17T13:59:05,373][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused>}
logstash       | [2023-01-17T13:59:05,379][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused"}
logstash       | [2023-01-17T13:59:05,436][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused>}
logstash       | [2023-01-17T13:59:05,444][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash       | [2023-01-17T13:59:05,449][WARN ][logstash.licensechecker.licensereader] Attempt to validate Elasticsearch license failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused"}
logstash       | [2023-01-17T13:59:05,477][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
logstash       | [2023-01-17T13:59:05,567][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash       | [2023-01-17T13:59:05,661][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/home/ettore/Documenti/Portfolio/ELK/logstash/logstash.conf"}
logstash       | [2023-01-17T13:59:05,664][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
logstash       | [2023-01-17T13:59:06,333][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash       | [2023-01-17T13:59:06,411][INFO ][logstash.runner          ] Logstash shut down.
logstash       | [2023-01-17T13:59:06,419][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
logstash       | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash       |    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
logstash       |    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
logstash       |    at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

and this is to prove that everything is working as intended with es (or so it seems)

netstat -an | grep 9200

tcp        0      0 0.0.0.0:9200            0.0.0.0:*               LISTEN     
tcp6       0      0 :::9200                 :::*                    LISTEN     
unix  3      [ ]         STREAM     CONNECTED     49200  

I've looked through everything and this is 100% not a duplicate because I have tried it all. I really can't figure it out. Hope anyone can help.

Thank you for you time.

Answer 1

You should set logstash.yml

1. Create a logstash.yml with values below:

http.host: "0.0.0.0"

xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]

2. In your docker-compose.yml, add another volume in Logstash container as shown below:

./logstash.yml:/usr/share/logstash/config/logstash.yml

Additionally, its good to run with restart condition.