Grafana 的 Elasticsearch 数据源不工作

Question

I have Grafana working with Prometheus as default datasource. Want to add Elasticsearch as a secondary data source. My environment is running in kubernetes with Elasticsearch: 7.2.0, Grafana: 6.4.4

But getting error No date field named undefined found

I am able to curl 'http://elasticsearch:9200/_cat/indices?v&pretty' from Grafana pod

root@shell-545fcb88cc-grn9t:/# curl -XGET 'http://elasticsearch:9200/_cat/indices?v&pretty'
health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   syslog-2019.11.03                  hfPuNniuTaKiKm6LZYazCg   1   1        277            0    386.9kb        193.4kb
green  open   syslog-2019.11.05                  JutF5UINTJaencoQ2TDllg   1   1        636            0    664.5kb        332.2kb
green  open   syslog-2019.10.31                  PynRH8ZMT0OwwpEKOvJgwg   1   1         42            0    164.1kb           82kb
green  open   filebeat-7.2.0-2019.10.03-000002   hhi_PGsaRWu9W2Lb565cyg   1   1   24963987            0     20.2gb         10.1gb
green  open   metricbeat-7.2.1-2019.10.03-000002 2ek3X0h-R5ONjD51sVcOCg   1   1    2757322            0        1gb        552.9mb
green  open   syslog-2019.11.02                  e9mcQ3KYS3yfegO7i6pn9g   1   1        319            0    475.4kb        237.7kb
green  open   metricbeat-7.2.0-2019.10.22-000003 ssCSFuT0QaikLW8OiKP1Kw   1   1  120115580            0     79.2gb         39.6gb
green  open   syslog-2019.11.07                  PcdvTxMRTnqS0ZUBlyK1rA   1   1       8538            0        5mb          2.5mb
green  open   filebeat-7.2.0-2019.11.02-000003   4MIsjbyKQ52DVxOUxeM9AQ   1   1    6350545            0      4.7gb          2.3gb
green  open   filebeat-7.2.0-2019.09.03-000001   eYZWZhv0RjybEkkUImJIiQ   1   1   25802336            0     21.3gb         10.6gb
green  open   syslog-2019.10.30                  e6m-tWt2S7S5fx739HIvCA   1   1       1143            0    823.7kb          419kb
green  open   metricbeat-7.2.1-2019.11.02-000003 iVEXirNxTJWbz8aCmpDENA   1   1          0            0       566b           283b
green  open   filebeat-7.2.1-2019.09.03-000001   I8vRGDnrR-6Cl5oPwCzjuw   1   1    4651714            0      3.4gb          1.7gb
green  open   metricbeat-7.2.0-2019.09.03-000001 abOCdQ4WTS62M75gpG6qkA   1   1  149807959            0      100gb           50gb
green  open   metricbeat-7.2.0-2019.09.28-000002 uYvCxqH7SpSKDYDdF1z_Rg   1   1  151770825            0      100gb           50gb
green  open   syslog-2019.11.06                  9onA0R_qTTKeroZvGlKCIg   1   1       3098            0      2.6mb          1.3mb
green  open   metricbeat-7.2.1-2019.09.03-000001 ZQ3-urydTn2kMhYDPtRqYQ   1   1    8700321            0      3.7gb          1.8gb
green  open   filebeat-7.2.1-2019.10.03-000002   Xkwv7LQVSm6yVqBSZKVKCg   1   1     389496            0    285.6mb        143.2mb
green  open   filebeat-7.2.1-2019.11.02-000003   5PwPy8jRQEC3bmW9hFctxg   1   1          0            0       566b           283b
green  open   syslog-2019.11.01                  wFgiLIxgQ9m4R8c1plR6UQ   1   1        943            0      933kb        457.6kb
green  open   syslog-2019.11.04                  OoaAXVTwRhiJfMlCFqI27Q   1   1       1469            0      1.3mb        713.4kb

Data Sources / Elasticsearch

HTTP

URL: http://elasticsearch:9200

Access: Server (Default)

Auth

No auth is selected.

Elasticsearch details

Index name: metricbeat-7.2.0-2019.10.22-000003

Pattern: No pattern

Time field name: @timestamp

Version: 7.0+

Max concurrent Shard Requests: 5

Min time interval: 10s

Logs

Message field name: (Blank)

Level field name: (Blank)

For now I just want to get metricbeat logs. How can I get this working, Also can I specify wildcard index name?

Answer 1

Your Elasticsearch datasource configuration is a problem, because it covers only 1 index from one day:

Index name: metricbeat-7.2.0-2019.10.22-000003
Pattern: No pattern

Index names are very variable:

metricbeat-7.2.0-2019.09.03-000001
metricbeat-7.2.0-2019.09.28-000002
metricbeat-7.2.0-2019.10.22-000003
metricbeat-7.2.1-2019.09.03-000001
metricbeat-7.2.1-2019.10.03-000002
metricbeat-7.2.1-2019.11.02-000003

So your index names don't have generic names. I recommend to configure custom index name in beat configuration:

output.elasticsearch.index: "metricbeat-%{+yyyy.MM.dd}"

Then it can be configured in Grafana as:

Index name: [metricbeat-]YYYY.MM.DD
Pattern: Daily

Eventually, you can create index alias, which includes all current index name variations and you will use that alias in the Grafana configuration.