如何使用 Elastic Search 和 Statsd 配置 Kibana

Question

I am trying to do some metric calculation and store the value of the metric in ElasticSearch and view them via Kibana

I followed this tutorial https://www.digitalocean.com/community/tutorials/how-to-configure-statsd-to-collect-arbitrary-stats-for-graphite-on-ubuntu-14-04

and it helped me to set up StatsD with graphite UI, Carbon and Whiper as Back-end.

Now I want to use StatsD with ElasticSearch and Kibana as Visualization tool.

The issue I am facing is that Kibana has its own aggregation technique and I have already done so using statsd, eg: uniq counts. So I just want to plot the data which I store in Elastic Search and not aggregate it further, Is this possible to view data without aggregating them in Kibana.?

Kindly Help.

Thanks.

Answer 1

You can do this via statsd(client) + metricbeat(statsd plugin that runs daemon) and Elasticsearch + Kibana. Following steps:

set up elasticsearch : https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

vi /etc/yum.repos.d/elasticsearch.repo and input:

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

and install elasticsearch from above configured repo:

sudo yum install --enablerepo=elasticsearch elasticsearch

set up (systemctl) service:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

which can be started/stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

similarly, set up kibana, by just following : https://www.elastic.co/guide/en/kibana/current/rpm.html

vi /etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

and install kibana : sudo yum install kibana You can similarly enable, start/stop systemd services for kibana also.

I have the elasticsearch and kibana co-located, on same single node set up.

Now, just use a statsd client, in a flask application for example:

from flask import Flask
from elasticapm.contrib.flask import ElasticAPM
from statsd import StatsClient
from random import randint
from datetime import datetime
import time

statsd = StatsClient(host='0.0.0.0',port=8125,prefix='test')
app = Flask(__name__)
apm = ElasticAPM(app)

@app.route('/')
def hello():
    bfr = time.time()
    time.sleep(3)
    statsd.incr('baz',2)
    statsd.decr('ban')
    statsd.gauge('foo',-3,delta=True)
    statsd.set('users',randint(0,100))
    statsd.timing('req',int((time.time() - bfr)*1000))
    return "Hello World!"

This sets up our statsd client, for statsd server, just go to : http://<es-ip>:5601/app/kibana#/home/tutorial/statsdMetrics and follow up instructions.

Set the metricbeat, enable statsd on the same server as flask application. steps are:

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.8.1-x86_64.rpm sudo rpm -vi metricbeat-7.8.1-x86_64.rpm

modify output.elasticsearch section in /etc/metricbeat/metricbeat.yml :

sudo metricbeat modules enable statsd

and setup things ie metricbeat / statsd:

sudo metricbeat setup
sudo service metricbeat start

You can do a netstat -tulpn | grep -i 8125 netstat -tulpn | grep -i 8125 and see a statsd daemon running.

Next, just go to "dev tools" kibana section

and fire following query to check the statsd metrics : http://<es-ip>:5601/app/kibana#/dev_tools/console

GET /_search
{"query":{"bool":{"filter":{"term":{"event.module":"statsd"}}}},"size":10}

You can now go, create a dashboard, add visualizations with query filters 'event.module' is 'statsd', and even search for terms like .... eg accounts.authentication.login.num_users