Send Kubernetes cluster logs to AWS Elasticsearch

Question

I have a testing Kubernetes cluster and I created elasticsearch on AWS which include Kibana for the log management.

Endpoint: https://search-this-is-my-es-wuktx5la4txs7avvo6ypuuyri.ca-central-1.es.amazonaws.com

As far as I googled, I have to send logs from fluentd. Then I tried to implement DaemonSet using this article . No luck.

Could you please share any good documentation to me, please

Answer 1

Kibana provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

To push log data into Elasticsearch, mostly people uses logstash/fluentd(log/data collectors)

Checkout below links for more info:

https://www.elastic.co/webinars/introduction-elk-stack

https://logz.io/blog/fluentd-logstash/

Answer 2

I had a similar problem. Below are the full details of how I got it working.

Setup:

• AWS ES instance accessible via a VPC.
• Using this yaml file as a template.
• k8s client version v1.9.2
• k8s server version v1.8.7

Host problem:

The main problem I had was with defining the environment variables correctly. For FLUENT_ELASTICSEARCH_HOST , I was including the https:// prefix on the host URL. Once I removed that, my connection problems went away.

Authentication:

There's no username or password configured for AWS ES. Per this discussion , I set the FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD values to null.

Sample configuration:

Here's the full set of environment variables in my daemonset yaml file:

- name:  FLUENT_ELASTICSEARCH_HOST
  value: "vpc-MY-DOMAIN.REGION.es.amazonaws.com"
- name:  FLUENT_ELASTICSEARCH_PORT
  value: "443"
- name: FLUENT_ELASTICSEARCH_SCHEME
  value: "https"
- name: FLUENT_ELASTICSEARCH_USER
  value: null
- name: FLUENT_ELASTICSEARCH_PASSWORD
  value: null

Bonus: connecting to Kibana

Instead of setting up AWS Cognito, I created an nginx pod in my kubernetes cluster that I use as a proxy to reach Kibana. I use the kubectl port-foward command to reach the nginx server from my local machine.

Here's my nginx.conf:

server {
  listen 80;
  listen [::]:80;

  server_name MY-DOMAIN;

  location /_plugin/kibana {
      proxy_pass https://vpc-MY-DOMAIN.REGION.es.amazonaws.com/_plugin/kibana;
  }
  location / {
      proxy_pass https://vpc-MY-DOMAIN.REGION.es.amazonaws.com;
  }
}

Once the nginx pod is deployed, I run this command:

kubectl port-forward POD_NAME 8888:80

Now the Kibana is accessible at http://localhost:8888/_plugin/kibana

I'm still having a timeout issue with the port-foward command and a problem with nginx caching the ES service IP (since that can change), but I'll update my response once I resolve those issues.