Can't connect elasticsearch and kibana on kubernetes single mode

Question

I try to run elasticsearch and kibana on kubernetes. I ran:

kubectl run elasticsearch --image=elasticsearch:6.6.1 --env="discovery.type=single-node" --port=9200 --port=9300
kubectl run kibana --image=kibana:6.6.1 --port=5601

then I ran $kubectl proxy ,

http://localhost:$IP_FROM_KUBECTL_PROXY(usually 8081)/api/v1/namespaces/default/pods/$POD_NAME/proxy/

When I entered to elasticsearch pod, everything looks fine, but when I entered to kibana, the app didn't work (I see "Kibana server is not ready yet" for infinity).

The logs of kibana are the following:

{"type":"log","@timestamp":"2019-03-02T10:38:47Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"log","@timestamp":"2019-03-02T10:38:49Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}

This is kibana.yml on kibana pod:

Default Kibana configuration from kibana-docker.

server.name: kibana
server.host: "0"
elasticsearch.url: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

I'm pretty new with Kubernetes, and I can't figure out why they can't talk one to another.

Answer 1

This Kibana log entry explains to you what the problem is:

{"type":"log","@timestamp":"2019-03-02T10:38:49Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}

Problem: Naming your pod elasticsearch is not enough for kubernetes.

You have a few solutions to fix it depending on the situation:

1. Create a service as Amityo has suggested. This is enough if kibana and elasticsearch are running in the same namespace.

2. If kibana and elasticsearch are running in different namespaces you'll need to use the full DNS name for service: elasticsearch.my-namespace.svc.cluster.local

3. If you run elasticsearch and kibana in the same pod. Then localhost:9200 will be enough to be able to query.

4. For your current situation. When elasticsearch is running you can use as ELASTICSEARCH_URL the pod DNS name: 1-2-3-4.default.pod.cluster.local when 1-2-3-4 is IP address of pod with dots replaced by dashes.

You can define the hostname when you create a Pod definition for elasticsearch using the following YAML:

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch
  labels:
    name: elasticsearch-single
spec:
  hostname: elasticsearch
  subdomain: for-kibana
  containers:
  - image: elasticsearch:6.6.1
    name: elasticsearch

Then you will be able to use as ELASTICSEARCH_URL the pod DNS name: elasticsearch.for-kibana.default.svc.cluster.local service.

All information you can find on kubernetes doc here

Please note: According to official docs the Env variable ELASTICSEARCH_URL is deprecated in newer elasticsearch versions, more details see here

Answer 2

In kubernetes pods communicate with Services. You will need to define a service that selects your pod (with selector).

for example:

kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
spec:
  selector:
    app: elasticsearch
  ports:
  - protocol: TCP
    port: 9200
    targetPort: 9200

Read more about services here and about labels here

We usually define the pods as yml files and add the labels there, but if you want to use kubectl run you can add labels with -l

  -l, --labels='': Comma separated labels to apply to the pod(s). Will override previous values.