elasticsearch on kubernetes - discovery of nodes

Question

We are attempting to run Elasticsearch on top of a kubernetes / flannel / coreos cluster.

As flannel does not support multicast , we cannot use Zen multicast discovery to allow the nodes to find each other, form a cluster and communicate.

Short of hard-coding the IP addresses of all the kubernetes nodes into the ES-config-file, is there another method we can utilise to assist in discovery? Possibly using etcd2 or some other kubernetes-compatible discovery service?

Answer 1

There is a discovery plugin that uses the kubernetes API for cluster discovery:

https://github.com/fabric8io/elasticsearch-cloud-kubernetes

Install the plugin:

/usr/share/elasticsearch/bin/plugin -i io.fabric8/elasticsearch-cloud-kubernetes/1.3.0 --verbose

Create a Kubernetes service for discovery:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-cluster
spec:
  ports:
    - port: 9300
  selector:
    app: elasticsearch

And an elasticsearch.yml :

cloud.k8s.servicedns: elasticsearch-cluster
discovery.type: io.fabric8.elasticsearch.discovery.k8s.K8sDiscoveryModule

Answer 2

Place the containers into a Kubernetes Service. The Kubernetes API makes an 'endpoints' API available that lists the IP addresses of all of the members of a service. This endpoint set will dynamically shrink and grow as you scale the number of pods.

You can access endpoints with:

kubectl get endpoints <service-name>

or directly via the Kubernetes API, see:

https://github.com/kubernetes/kubernetes/blob/master/examples/cassandra/java/src/io/k8s/cassandra/KubernetesSeedProvider.java#L106

for an example of how this was done for Cassandra.

Answer 3

Version 6.2.0 is supporting kubernetes auto discovery

update your elasticsearch.yml as following

discovery.zen.ping.unicast.hosts: "kubernetes service name"

Answer 4

It worked for me only in this configuration.
Important! flannel must be enabled with vxlan.

cluster.yaml

network:
  plugin: flannel
  options:
    flannel_backend_type: vxlan

elasticsearch.yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-cluster
spec:
  version: 7.0.1
  nodeSets:
    - name: node
      count: 3
      config:
        node.master: true
        node.data: true
        node.ingest: true
        xpack.ml.enabled: true
        node.store.allow_mmap: true
        indices.query.bool.max_clause_count: 100000

#        Fixed flannel kubernetes network plugin
        discovery.seed_hosts:
        {{ range $i, $e := until (3 | int) }}
          - elastic-cluster-es-node-{{ $i }}
        {{ end }}

      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms4g -Xmx4g"
                - name: READINESS_PROBE_TIMEOUT
                  value: "60"
              resources:
                requests:
                  memory: 5Gi
#                  cpu: 1
                limits:
                  memory: 6Gi
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            storageClassName: local-elasticsearch-storage
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 5G