如何测试 ElasticSearch Logstash 和 Kibana

Question

I have installed elasticsearch, logstash and kibana to my Debian server. The problem is Kibana is not showing any statistics or logs. I don't know what is wrong and how to debug this problem. When I test each of the components (elasticsearch, kibana and logstash) everything looks working properly.

ElasticSearch Tests

• Checking elasticsearch-cluster status:

curl 'localhost:9200/_cluster/health?v'

{"cluster_name":"elasticsearch","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":71,"active_shards":71,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":71,"number_of_pending_tasks":0}

• Checking elasticsearch-node status:

curl 'localhost:9200/_cat/nodes?v'

host ip            heap.percent ram.percent load node.role master name    
ais  193.xx.yy.zz            6      10     0.05      d       *   Shathra

• Checking elasticsearch-index status:

curl 'localhost:9200/_cat/indices?v'

health status index              pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   countries            5   1        243          365    145.2kb        145.2kb 
yellow open   imports              5   1         26            7     49.6kb         49.6kb 
yellow open   categories           5   1          6            1     20.6kb         20.6kb 
yellow open   faculties            5   1         36            0     16.9kb         16.9kb 
yellow open   users                5   1       6602           29      1.8mb          1.8mb 
yellow open   cities               5   1        125            0     23.5kb         23.5kb 
yellow open   exam_languages       5   1        155            0     26.6kb         26.6kb 
yellow open   departments          5   1        167           70    166.4kb        166.4kb 
yellow open   examinations         5   1          4            0     14.1kb         14.1kb 
yellow open   certificates         5   1          1            0        3kb            3kb 
yellow open   .kibana              1   1          2            1       14kb           14kb 
yellow open   exam_centers         5   1          5            0     22.7kb         22.7kb 

• Checking elasticsearch-service status:

$ service elasticsearch status

[ ok ] elasticsearch is running.

ElasticSearch is also reacable from localhost:9200 in my browser and listing indexes correct.

/etc/nginx/sites-available/elasticsearch file =>

server {
  listen 443;
  server_name es.xxx.yyy.com;
  ssl on;
  ssl_certificate /etc/elasticsearch/ssl/es_domain.crt;
  ssl_certificate_key /etc/elasticsearch/ssl/es_domain.key;
  access_log /var/log/nginx/elasticsearch/access.log;
  error_log /var/log/nginx/elasticsearch/error.log debug;
  location / {
    rewrite ^/(.*) /$1 break;
    proxy_ignore_client_abort on;
    proxy_pass http://localhost:9200;
    proxy_redirect http://localhost:9200 http://es.xxx.yyy.com/;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $http_host;
    auth_basic "Elasticsearch Authentication";
    auth_basic_user_file /etc/elasticsearch/user.pwd;
  }
}

server{
  listen 80;
  server_name es.xxx.yyy.com;
  return 301 https://$host$request_uri;
}

Kibana Tests

$ service kibana4 status

[ ok ] kibana is running.

/etc/nginx/sites-available/kibana file =>

server {
  listen 443;
  server_name kibana.xxx.yyy.com;
  ssl on;
  ssl_certificate /opt/kibana/ssl/es_domain.crt;
  ssl_certificate_key /opt/kibana/ssl/es_domain.key;
  access_log /var/log/nginx/kibana/access.log;
  error_log /var/log/nginx/kibana/error.log debug;
  location / {
    rewrite ^/(.*) /$1 break;
    proxy_ignore_client_abort on;
    proxy_pass http://localhost:5601;
    proxy_redirect http://localhost:5601 http://kibana.xxx.yyy.com/;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $http_host;
    auth_basic "Kibana Authentication";
    auth_basic_user_file /etc/nginx/htpasswd.users;
  }
}

server{
  listen 80;
  server_name kibana.xxx.yyy.com;
  return 301 https://$host$request_uri;
}

Kibana is also reacable from localhost:5601 in my browser without any problem.

Logstash Tests

$ sudo /etc/init.d/logstash status

[ ok ] logstash is running.

/etc/logstash/conf.d/01-ais-input.conf file =>

input {
  file {
    type => "rails"
    path => "/srv/www/xxx.yyy.com/site/log/logstasher.log" 
    codec => json {
      charset => "UTF-8"
    }
  }
}

output {
  elasticsearch {
   host => 'localhost'
   port => 9200
  }
}

Is there anything wrong with these services and config files? Each of the components look working fine but I can not see anything in Kibana interface. How can I test my ELK stack?

Answer 1

You need to configure index patterns in Kibana to see the elasticsearch data.

1. Open Kibana from your browser http://localhost:5601
2. Click on Settings
3. Type your existing index name and click Create. (Uncheck the option 'Index contains time-based events' unless your index is having logs or any time-stamp based data)

Doing this, you must be able to see all your elasticsearch documents.