GKE中Elasticsearch的StackDriver监视

Question

我需要监视安装在k8s集群顶部的Elasticsearch（2.4）。 我有2个客户端，3个主节点和几个在pod中运行的数据节点。 遵循Stackdriver的“操作方法”和“ 我可以在Kubernetes Pod中运行Google Monitoring Agent吗？ ”一文之后，我在自己的Pod中部署了一个代理。 毕竟，突然之间，StackDriver中没有Elasticsearch指标。 唯一的零。

任何建议都值得欢迎。

这是我的配置：

弹性服务：

$kubectl describe svc elasticsearch
Name:           elasticsearch
Namespace:      default
Labels:         component=elasticsearch
            role=client
Selector:       component=elasticsearch,role=client
Type:           NodePort
IP:         <IP>
Port:           http    9200/TCP
NodePort:       http    <PORT>/TCP
Endpoints:      <IP>:9200,<IP>:9200
Session Affinity:   None
No events.

Stackdriver部署：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: stackagent
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: monitoring
        role: stackdriver-agent
    spec:
      containers:
      - name: hslab-data-agent
        image: StackDriverAgent:version1

StackDriverAgent：version1 Docker：

FROM ubuntu

WORKDIR /stackdriver

RUN apt-get update
RUN apt-get install curl lsb-release libyajl2 -y
RUN apt-get clean

COPY ./stackdriver/run.sh run.sh
COPY ./stackdriver/elasticsearch.conf elasticsearch.conf

RUN chmod 755 ./run.sh
CMD ["./run.sh"]

run.sh：

#!/bin/bash

curl -O https://repo.stackdriver.com/stack-install.sh

chmod 755 stack-install.sh
bash stack-install.sh --write-gcm

cp ./elasticsearch.conf /opt/stackdriver/collectd/etc/collectd.d/

service stackdriver-agent restart

while true; do
    sleep 60
    agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null)

    ps -p $agent_pid > /dev/null 2>&1
    if [ $? != 0 ]; then
        echo "Stackdriver agent pid not found!"
        break;
    fi
done

elasticsearch.conf：

取自https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/elasticsearch.conf

# This is the monitoring configuration for Elasticsearch 1.0.x and later.
# Look for ELASTICSEARCH_HOST and ELASTICSEARCH_PORT to adjust your configuration file.
LoadPlugin curl_json
<Plugin "curl_json">
    # When using non-standard Elasticsearch configurations, replace the below with
    #<URL "http://ELASTICSEARCH_HOST:ELASTICSEARCH_PORT/_nodes/_local/stats/">
    # PREVIOUSE LINE
    # <URL "http://localhost:9200/_nodes/_local/stats/"> 
    <URL "http://elasticsearch:9200/_nodes/_local/stats/">
        Instance "elasticsearch"
 ....

运行状态：

NAME                                      READY     STATUS    RESTARTS   AGE
esclient-4231471109-bd4tb                 1/1       Running   0          23h
esclient-4231471109-k5pnw                 1/1       Running   0          23h
esdata-1-2524076994-898r0                 1/1       Running   0          23h
esdata-2-2426789420-zhz7j                 1/1       Running   0          23h
esmaster-1-4205943399-zj2pn               1/1       Running   0          23h
esmaster-2-4248445829-pwq46               1/1       Running   0          23h
esmaster-3-3967126695-w0tp2               1/1       Running   0          23h
stackagent-3122989159-15vj1               1/1       Running   0          18h

Answer 1

插件配置的API URL问题。http：// elasticsearch：9200 / _nodes / ** _ local ** / stats /“>将仅返回_local节点的信息，该节点是没有文档的客户端。

另外，Stackdriver数据将在k8s群集节点下收集，而不是在Pod名称下收集。

部分解决方案是在数据节点中设置sidecar，并使用对应的ES节点名称修补elasticsearch.conf查询：

1. curl [elasticsearch]:9200/_nodes/stats
2. 通过$（hostname）查找ES节点名称
3. 修补配置<URL "http://elasticsearch:9200/_nodes/<esnode_name>/stats/">

这将在k8s数据节点名称下收集ES数据节点的信息。