GKE中Elasticsearch的StackDriver監視

Question

我需要監視安裝在k8s集群頂部的Elasticsearch（2.4）。 我有2個客戶端，3個主節點和幾個在pod中運行的數據節點。 遵循Stackdriver的“操作方法”和“ 我可以在Kubernetes Pod中運行Google Monitoring Agent嗎？ ”一文之后，我在自己的Pod中部署了一個代理。 畢竟，突然之間，StackDriver中沒有Elasticsearch指標。 唯一的零。

任何建議都值得歡迎。

這是我的配置：

彈性服務：

$kubectl describe svc elasticsearch
Name:           elasticsearch
Namespace:      default
Labels:         component=elasticsearch
            role=client
Selector:       component=elasticsearch,role=client
Type:           NodePort
IP:         <IP>
Port:           http    9200/TCP
NodePort:       http    <PORT>/TCP
Endpoints:      <IP>:9200,<IP>:9200
Session Affinity:   None
No events.

Stackdriver部署：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: stackagent
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: monitoring
        role: stackdriver-agent
    spec:
      containers:
      - name: hslab-data-agent
        image: StackDriverAgent:version1

StackDriverAgent：version1 Docker：

FROM ubuntu

WORKDIR /stackdriver

RUN apt-get update
RUN apt-get install curl lsb-release libyajl2 -y
RUN apt-get clean

COPY ./stackdriver/run.sh run.sh
COPY ./stackdriver/elasticsearch.conf elasticsearch.conf

RUN chmod 755 ./run.sh
CMD ["./run.sh"]

run.sh：

#!/bin/bash

curl -O https://repo.stackdriver.com/stack-install.sh

chmod 755 stack-install.sh
bash stack-install.sh --write-gcm

cp ./elasticsearch.conf /opt/stackdriver/collectd/etc/collectd.d/

service stackdriver-agent restart

while true; do
    sleep 60
    agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null)

    ps -p $agent_pid > /dev/null 2>&1
    if [ $? != 0 ]; then
        echo "Stackdriver agent pid not found!"
        break;
    fi
done

elasticsearch.conf：

取自https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/elasticsearch.conf

# This is the monitoring configuration for Elasticsearch 1.0.x and later.
# Look for ELASTICSEARCH_HOST and ELASTICSEARCH_PORT to adjust your configuration file.
LoadPlugin curl_json
<Plugin "curl_json">
    # When using non-standard Elasticsearch configurations, replace the below with
    #<URL "http://ELASTICSEARCH_HOST:ELASTICSEARCH_PORT/_nodes/_local/stats/">
    # PREVIOUSE LINE
    # <URL "http://localhost:9200/_nodes/_local/stats/"> 
    <URL "http://elasticsearch:9200/_nodes/_local/stats/">
        Instance "elasticsearch"
 ....

運行狀態：

NAME                                      READY     STATUS    RESTARTS   AGE
esclient-4231471109-bd4tb                 1/1       Running   0          23h
esclient-4231471109-k5pnw                 1/1       Running   0          23h
esdata-1-2524076994-898r0                 1/1       Running   0          23h
esdata-2-2426789420-zhz7j                 1/1       Running   0          23h
esmaster-1-4205943399-zj2pn               1/1       Running   0          23h
esmaster-2-4248445829-pwq46               1/1       Running   0          23h
esmaster-3-3967126695-w0tp2               1/1       Running   0          23h
stackagent-3122989159-15vj1               1/1       Running   0          18h

Answer 1

插件配置的API URL問題。http：// elasticsearch：9200 / _nodes / ** _ local ** / stats /“>將僅返回_local節點的信息，該節點是沒有文檔的客戶端。

另外，Stackdriver數據將在k8s群集節點下收集，而不是在Pod名稱下收集。

部分解決方案是在數據節點中設置sidecar，並使用對應的ES節點名稱修補elasticsearch.conf查詢：

1. curl [elasticsearch]:9200/_nodes/stats
2. 通過$（hostname）查找ES節點名稱
3. 修補配置<URL "http://elasticsearch:9200/_nodes/<esnode_name>/stats/">

這將在k8s數據節點名稱下收集ES數據節點的信息。