StackDriver Monitoring for Elasticsearch in GKE

Question

I need to monitor Elasticsearch(2.4) that installed on top of the k8s cluster. I have 2 clients, 3 masters and several data nodes run in pods. Following the "how to" of Stackdriver and the post " Can I run Google Monitoring Agent inside a Kubernetes Pod? ", I deployed an agent in its own Pod. Suddenly, after all, have no Elasticsearch metrics in StackDriver. The Only Zeros.

Any suggestion are more than welcome.

This is my configuration:

Elastic service:

$kubectl describe svc elasticsearch
Name:           elasticsearch
Namespace:      default
Labels:         component=elasticsearch
            role=client
Selector:       component=elasticsearch,role=client
Type:           NodePort
IP:         <IP>
Port:           http    9200/TCP
NodePort:       http    <PORT>/TCP
Endpoints:      <IP>:9200,<IP>:9200
Session Affinity:   None
No events.

Stackdriver deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: stackagent
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: monitoring
        role: stackdriver-agent
    spec:
      containers:
      - name: hslab-data-agent
        image: StackDriverAgent:version1

StackDriverAgent:version1 Docker:

FROM ubuntu

WORKDIR /stackdriver

RUN apt-get update
RUN apt-get install curl lsb-release libyajl2 -y
RUN apt-get clean

COPY ./stackdriver/run.sh run.sh
COPY ./stackdriver/elasticsearch.conf elasticsearch.conf

RUN chmod 755 ./run.sh
CMD ["./run.sh"]

run.sh:

#!/bin/bash

curl -O https://repo.stackdriver.com/stack-install.sh

chmod 755 stack-install.sh
bash stack-install.sh --write-gcm

cp ./elasticsearch.conf /opt/stackdriver/collectd/etc/collectd.d/

service stackdriver-agent restart

while true; do
    sleep 60
    agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null)

    ps -p $agent_pid > /dev/null 2>&1
    if [ $? != 0 ]; then
        echo "Stackdriver agent pid not found!"
        break;
    fi
done

elasticsearch.conf:

Taken from https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/elasticsearch.conf

# This is the monitoring configuration for Elasticsearch 1.0.x and later.
# Look for ELASTICSEARCH_HOST and ELASTICSEARCH_PORT to adjust your configuration file.
LoadPlugin curl_json
<Plugin "curl_json">
    # When using non-standard Elasticsearch configurations, replace the below with
    #<URL "http://ELASTICSEARCH_HOST:ELASTICSEARCH_PORT/_nodes/_local/stats/">
    # PREVIOUSE LINE
    # <URL "http://localhost:9200/_nodes/_local/stats/"> 
    <URL "http://elasticsearch:9200/_nodes/_local/stats/">
        Instance "elasticsearch"
 ....

Running state:

NAME                                      READY     STATUS    RESTARTS   AGE
esclient-4231471109-bd4tb                 1/1       Running   0          23h
esclient-4231471109-k5pnw                 1/1       Running   0          23h
esdata-1-2524076994-898r0                 1/1       Running   0          23h
esdata-2-2426789420-zhz7j                 1/1       Running   0          23h
esmaster-1-4205943399-zj2pn               1/1       Running   0          23h
esmaster-2-4248445829-pwq46               1/1       Running   0          23h
esmaster-3-3967126695-w0tp2               1/1       Running   0          23h
stackagent-3122989159-15vj1               1/1       Running   0          18h

Answer 1

The problem with API URL of plugin configuration.The http://elasticsearch:9200/_nodes/**_local**/stats/"> will returns info only for the _local node which is a client that have no documents.

In addition, the Stackdriver data will be collected under k8s cluster node and not under the pod name.

The partial solution is to setup sidecar in data node and patch the elasticsearch.conf query with the corresponding ES node name:

1. get the curl [elasticsearch]:9200/_nodes/stats
2. find the ES node name by the $(hostname)
3. patch the configuration <URL "http://elasticsearch:9200/_nodes/<esnode_name>/stats/">

This will collect the information of ES data node under the k8s data node name.