Kibana not showing monitoring data from external Elasticsearch node

Question

yesterday I setup a dedicated single monitoring node following this guide.

I managed to fire up the new monitoring node with the same ES 6.6.0 version of the cluster, then added those lines to my elasticsearch.yml file on all ES cluster nodes :

xpack.monitoring.exporters:
  id1:
 type: http
  host: ["http://monitoring-node-ip-here:9200"]

Then restarted all nodes and Kibana (that is actually running in one of the node of the ES cluster).

Now I can see today monitoring data indices being sent to the new monitoring external node but Kibana is showing a " You need to make some adjustments " when accessing the "Monitoring" section.

We checked the  `cluster defaults`  settings for  `xpack.monitoring.exporters` , and found the 
reason:  `Remote exporters indicate a possible misconfiguration: id1`

Check that the intended exporters are enabled for sending statistics to the monitoring cluster, 
and that the monitoring cluster host matches the  `xpack.monitoring.elasticsearch`  setting in  
`kibana.yml`  to see monitoring data in this instance of Kibana.

I already checked that all nodes are pingable each other , also I don't have xpack security so I haven't created any additional "remote_monitor" user.

I followed the error message and tried to add the xpack.monitoring.elasticsearch in kibana.yml file but I ended up with the following error :

 FATAL  ValidationError: child "xpack" fails because [child "monitoring" fails because [child 
"elasticsearch" fails because ["url" is not allowed]]]

Hope anyone can help me in figuring what's wrong.

Answer 1

EDIT #1

Solved : problem was due to monitoring not being disabled in the monitoring cluster :

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": false
  }
}

Additional I made a mistake in kibana.yml configuration, xpack.monitoring.elasticsearch should have been xpack.monitoring.elasticsearch.hosts

Answer 2

i had exactly the same problem but the root of cause was smth different.

here have a look

okay, i used to have the same problem.

my kibana did not show monitoring graphs, however i had monitoring index index .monitoring-es-* available

the root of problem in my case was that my master nodes did not have :9200 HTTP socket available from the LAN. that is my config on master nodes was:

...
transport.host: [ "192.168.7.190" ]
transport.port: 9300
http.port: 9200
http.host: [ "127.0.0.1" ]
...

as you can see HTTP socket is available only from within host.

i didnt want if some one will make HTTP request for masters from LAN because there is no point to do that.

However as i uderstand Kibana do not only read data from monitoring index

index .monitoring-es-*

but also make some requests directly for masters to get some information.

It was exactly why Kibana did not show anything about monitoring.

After i changed one line in the config on master node as

http.host: [ "192.168.0.190", "127.0.0.1" ]

immidiately kibana started to show monitoring graphs.

i recreated this expereminet several times.

Now all is working.

Also i want to underline in spite that now all is fine my monitoring index .monitoring-es-* do NOT have "cluster_stats" documents.

So if your kibana do not show monitoring graphs i suggest

1. check if index .monitoring-es-* exists
2. check if your master nodes can serve HTTP requests from LAN