Unable to retrieve version information from Elasticsearch nodes. Request timed out

Question

I am installing Kibana and elasticsearch version 7.15.1 as per instructions mentioned in the link Install Kibana with Docker

The commands I am using are

docker network create elastic
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.15.1
docker run -d --name es01-test --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.1

docker pull docker.elastic.co/kibana/kibana:7.15.1
docker run -d --name kib01-test --net elastic -p 5601:5601 -e "ELASTICSEARCH_HOSTS=http://es01-test:9200" docker.elastic.co/kibana/kibana:7.15.1
To access Kibana, go to http://localhost:5601.

I am receiving the response from elasticsearch

anshuman@anshuman:~$ curl http://localhost:9200
{
  "name" : "ac2f811ab55e",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "9HbzlMw0QFurysYIVYWfsA",
  "version" : {
    "number" : "7.15.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "83c34f456ae29d60e94d886e455e6a3409bba9ed",
    "build_date" : "2021-10-07T21:56:19.031608185Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

But I am receiving error from kibana

anshuman@anshuman:~$ curl http://localhost:5601
Kibana server is not ready yet

In the kibana logs, I see below error

{"type":"log","@timestamp":"2021-10-31T23:58:42+00:00","tags":["error","savedobjects-service"],"pid":1219,"message":"Unable to retrieve version information from Elasticsearch nodes. Request timed out"}

In the elasticsearch logs, I see below error

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Elasticsearch (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.1.jar)
WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Security (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.1.jar)
WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
{"type": "server", "timestamp": "2021-11-01T00:49:56,864Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "8666cedc7085", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-8550956835462737955, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms7798m, -Xmx7798m, -XX:MaxDirectMemorySize=4089446400, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2021-11-01T00:50:32,754Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "8666cedc7085", "message": "exception during geoip databases update", "cluster.uuid": "cFfg1C69SU20aq_Hdu-trA", "node.id": "PK7tnmg6Tk-icw0SElkK4A" , 

My System details are

anshuman@anshuman:~$ docker -v
Docker version 20.10.8, build 3967b7d28e
anshuman@anshuman:~$ uname -a
Linux anshuman 5.13.0-20-generic #20-Ubuntu SMP Fri Oct 15 14:21:35 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
anshuman@anshuman:~$ neofetch
 OS: Ubuntu 21.10 x86_64 
 Kernel: 5.13.0-20-generic 
 CPU: Intel i5-2520M (4) @ 3.200GHz 
 GPU: NVIDIA GeForce 610M 
 GPU: Intel 2nd Generation Core Processor Family 
 Memory: 10773MiB / 15596MiB 

Complete Kibana logs

Complete elasticsearch logs

any recommendations?

Answer 1

Your kibana service is missing information about elasticsearch user/password.

Few days ago I tryied to create minimalistic swarm stack and this is result:

docker-compose.yml

version: "3.8"
services:
  elasticsearch:
    image: elasticsearch:7.15.1
    volumes:
      - /opt/apps/elk/elasticsearch:/usr/share/elasticsearch/data
    environment:
      - cluster.name=elasticsearch
      - network.host=0
      - discovery.type=single-node
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.monitoring.collection.enabled=true

  kibana:
    image: kibana:7.15.1
    ports:
      - 5601:5601
    volumes:
      - /opt/apps/elk/kibana/kibana.yml:/opt/kibana/config/kibana.yml

  logstash:
    image: logstash:7.15.1
    ports:
      # Java gelf listener
      - 12201:12201
    volumes:
      - /opt/apps/elk/logstash:/usr/share/logstash/pipeline

kibana.yml

server.host: "0.0.0.0"
server.shutdownTimeout: "5s"

elasticsearch.hosts: [ "http://elasticsearch:9200" ]
elasticsearch.username: "kibana_system"
elasticsearch.password: "<TO-BE-CREATED>"

xpack.security.enabled: "true"
# generate custom password with min length 32
xpack.encryptedSavedObjects.encryptionKey: RANDOM-PASSWORD
xpack.reporting.encryptionKey: RANDOM-PASSWORD
xpack.security.encryptionKey: RANDOM-PASSWORD

# optional
xpack.reporting.capture.browser.chromium.disableSandbox: false
xpack.reporting.roles.enabled: false

Now you have to:

1. Start whole stack (eg docker stack deploy -c docker-compose.yml elk )

2. "Login" into running elasticsearch service (eg docker exec -it elk_elasticsearch.1....... bash )

3. Generate new passwords for internal users and save them

   bin/elasticsearch-setup-passwords auto --batch --url http://elasticsearch:9200

4. Stop whole elk stack (eg docker stack rm elk )

5. Edit kibana.yml elasticsearch.password value accordingly to generated password

6. Start whole stack again and login to server using http://SERVER-IP:5601/

PS: I know that this isn't the best way to create ELK stack but it is working without doing intermediate steps. Also in this example there is no HTTPS support as all my instances are running behind Traefik proxy (excluded)

Answer 2

I have to explicitly set the option com.docker.network.bridge.enable_icc for the bridge connection. Now it is working fine. I am sharing the final docker-compose.yml .

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.15.2
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.enable_icc: "true"