Impossible connect to elasticsearch in kubernetes(bare metal)

Question

I've set up elastic + kibana + metricbeat in local cluster. But the metricbeat can't connect to elastic:

ERROR   pipeline/output.go:100  Failed to connect to 
backoff(elasticsearch(http://elasticsearch:9200)): Get http://elasticsearch:9200: lookup 
elasticsearch on 10.96.0.10:53: no such host
2019-10-15T14:14:32.553Z    INFO    pipeline/output.go:93   Attempting to reconnect to 
backoff(elasticsearch(http://elasticsearch:9200)) with 10 reconnect attempt(s)
2019-10-15T14:14:32.553Z    INFO    [publisher] pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-10-15T14:14:32.553Z    INFO    [publisher] pipeline/retry.go:191     done
2019-10-15T14:14:32.553Z    INFO    [publisher] pipeline/retry.go:166   retryer: send wait signal to consumer
2019-10-15T14:14:32.553Z    INFO    [publisher] pipeline/retry.go:168     done
2019-10-15T14:14:32.592Z    WARN    transport/tcp.go:53 DNS lookup failure "elasticsearch": lookup elasticsearch on 10.96.0.10:53: no such host

In my cluster I use metalldb and ingress. I've set up ingress rules but it didnt help me.

Also I've noticed that the elk and the metricbeat have different namespaces in docs. I've tried make everywhere the same namespaces but it was unsuccesfully.

Below I've attached my yamls. Files for elastic/kibana and metricbeat I didn't attach because they have a lot of lines, I wrote only ref on them:

1. elastic/kibana - https://download.elastic.co/downloads/eck/1.0.0-beta1/all-in-one.yaml

2. metricbeat - https://raw.githubusercontent.com/elastic/beats/7.4/deploy/kubernetes/metricbeat-kubernetes.yaml

Maybe anybody know why it happens?

 **elastic config** -
 apiVersion: elasticsearch.k8s.elastic.co/v1beta1
 kind: Elasticsearch
 metadata:
 name: quickstart
 spec:
 version: 7.4.0
 nodeSets:
 - name: default
 count: 1
 config:
  node.master: true
  node.data: true
  node.ingest: true
  node.store.allow_mmap: false
 volumeClaimTemplates:
 - metadata:
    name: elasticsearch-data # note: elasticsearch-data must be the name of the Elasticsearch volume
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
    storageClassName: standard
http:
service:
  spec:
    type: LoadBalancer

---

**kibana config** -
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
name: quickstart
spec:
version: 7.4.0
count: 1
elasticsearchRef:
name: quickstart
http:
service:
  spec:
    type: LoadBalancer
tls:
  selfSignedCertificate:
    disabled: true

---

**ingress rules** -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
spec:
rules:
- http:
  paths:
  - path: /
    backend:
      serviceName: undemo-service
      servicePort: 80
  - path: /
    backend:
      serviceName: quickstart-kb-http
      servicePort: 80
  - path: /
    backend:
      serviceName: quickstart-es-http
      servicePort: 80

Answer 1

Just to be aware. Filebeat, metricbeats... runs under kube-system namespace.

If you run elastic on default namespace you should use elasticsearch.default as host in order to resolve your service properly.