如何使用 docker swarm 部署 elasticsearch？

Question

我創建了 3 個虛擬機使用docker-machine ，有：

NAME       ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER     ERRORS
cluster    -        virtualbox   Running   tcp://192.168.99.101:2376           v18.09.5   
cluster2   -        virtualbox   Running   tcp://192.168.99.102:2376           v18.09.5   
master     -        virtualbox   Running   tcp://192.168.99.100:2376           v18.09.5 

然后我在master中創建一個 docker swarm：

 docker-machine ssh master "docker swarm init ----advertise-addr 192.168.99.100"

在cluster和cluster2加入master ：

docker-machine ssh cluster "docker swarm join --advertise-addr 192.168.99.101 --token xxxx 192.168.99.100:2377"

docker-machine ssh cluster2 "docker swarm join --advertise-addr 192.168.99.102 --token xxxx 192.168.99.100:2377"

docker node ls信息：

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
r4a6y9wie4zp3pl4wi4e6wqp8     cluster             Ready               Active                                  18.09.5
sg9gq6s3k6vty7qap7co6eppn     cluster2            Ready               Active                                  18.09.5
xb6telu8cn3bfmume1kcektkt *   master              Ready               Active              Leader              18.09.5

有部署配置swarm.yml ：

version: "3.3"

services:
  elasticsearch:
    image: elasticsearch:7.0.0
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - cluster.name=elk
      - network.host=_eth1:ipv4_
      - network.bind_host=_eth1:ipv4_
      - network.publish_host=_eth1:ipv4_
      - discovery.seed_hosts=192.168.99.100,192.168.99.101
      - cluster.initial_master_nodes=192.168.99.100,192.168.99.101
      - bootstrap.memory_lock=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    networks:
      - backend
    deploy:
      mode: replicated
      replicas: 3
      #endpoint_mode: dnsrr
      restart_policy:
        condition: none
      resources:
        limits:
          cpus: "1.0"
          memory: "1024M"
        reservations:
          memory: 20M
networks:
  backend:
    # driver: overlay
    # attachable: true

我將 elasticsearch 圖像拉到虛擬機：

docker-machine ssh master "docker image pull elasticsearch:7.0.0"
docker-machine ssh cluster "docker image pull elasticsearch:7.0.0"
docker-machine ssh cluster2 "docker image pull elasticsearch:7.0.0"

在運行之前，我運行此命令修復一些 elasticearch 引導程序錯誤：

docker-machine ssh master "sudo sysctl -w vm.max_map_count=262144"
docker-machine ssh cluster "sudo sysctl -w vm.max_map_count=262144"
docker-machine ssh cluster2 "sudo sysctl -w vm.max_map_count=262144"

然后我運行`docker stack deploy -c swarm.yml es，elasticsearch 集群無法工作。

docker-machine ssh master
docker service logs es_elasticsearch -f 

展示：

es_elasticsearch.1.uh1x0s9qr7mb@cluster    | {"type": "server", "timestamp": "2019-04-25T16:28:47,143+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "e8dba5562417",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{e8dba5562417}{Jy3t0AAkSW-jY-IygOCjOQ}{z7MYIf5wTfOhCX1r25wNPg}{10.255.0.46}{10.255.0.46:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es_elasticsearch.2.swswlwmle9e9@cluster2    | {"type": "server", "timestamp": "2019-04-25T16:28:47,389+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "af5d88a04b42",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{af5d88a04b42}{zhxMeNMAQN2evKDlsA33qA}{fpYPTvJ6STmyqrgxlMkD_w}{10.255.0.47}{10.255.0.47:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es_elasticsearch.3.x8ouukovhh80@master    | {"type": "server", "timestamp": "2019-04-25T16:28:48,818+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "0e7e4d96b31a",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{0e7e4d96b31a}{Xs9966RjTEWvEbuj4-ySYA}{-eV4lvavSHq6JhoW0qWu6A}{10.255.0.48}{10.255.0.48:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }

我猜集群形成失敗可能是由於網絡配置錯誤。 我不知道如何修復它，我嘗試了很多次修改配置，失敗並再次失敗。

Answer 1

試試，這是有效的:) docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    hostname: "{{.Node.Hostname}}"
    environment:
      - node.name={{.Node.Hostname}}
      - cluster.name=my-cluster
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=node1,node2,node3
      - node.ml=false
      - xpack.ml.enabled=false
      - xpack.monitoring.enabled=false
      - xpack.security.enabled=false
      - xpack.watcher.enabled=false
      - bootstrap.memory_lock=false
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    deploy:
      mode: global
      endpoint_mode: dnsrr
      resources:
        limits:
          memory: 4G
  nginx:
    image: nginx:1.17.1-alpine
    ports:
      - 9200:9200
    deploy:
      mode: global
    command: |
      /bin/sh -c "echo '
      user nobody nogroup;
      worker_processes auto;
      events {
        worker_connections 1024;
      }
      http {
        client_max_body_size 4g;
        resolver 127.0.0.11 ipv6=off;
        server {
          listen *:9200;
          location / {
            proxy_set_header Connection keep-alive;
            set $$url http://elasticsearch:9200;
            proxy_pass $$url;
            proxy_set_header  Host $$http_host;
            proxy_set_header  X-Real-IP $$remote_addr;
            proxy_set_header  X-Forwarded-For $$proxy_add_x_forwarded_for;
          }
        }
      }' | tee /etc/nginx/nginx.conf && nginx -t && nginx -g 'daemon off;'"

volumes:
  elasticsearch-data:

Answer 2

由於群覆蓋網絡，嘗試手動指定所有特定 IP 和綁定是很棘手的。 相反，只需讓您的 ES 節點可發現並讓 Swarm 負責節點發現和通信。 為了使它們可被發現，我們可以使用可預測的名稱，例如 Swarm 節點主機名。

嘗試更改swarm.yml文件中的環境設置，如下所示：

    environment:
    - network.host=0.0.0.0
    - discovery.seed_hosts=elasticsearch #Service name, to let Swarm handle discovery
    - cluster.initial_master_nodes=master,cluster,cluster2 #Swarm nodes host names
    - node.name={{.Node.Hostname}} #To create a predictable node name

這當然假設我們已經知道群主機名，您在上面的屏幕截圖中指出了這一點。 如果不知道這些值，我們將無法找到一組可預測的節點名稱。 在這種情況下，您可以創建一個具有特定節點名稱的 ES 節點條目，然后創建另一個將第一個條目的節點名稱引用為cluster.initial_master_nodes的條目。

Answer 3

使用沒有端口的 dnsrr 模式。 使用 nginx 公開 elasticsearch；）查看我的 docker-compose.yml

Answer 4

根據我的經驗， https://github.com/shazChaudhry/docker-elastic工作得很好，整個 repo 中只有一個文件就足夠了。 我下載了https://github.com/shazChaudhry/docker-elastic/blob/master/docker-compose.yml並刪除了 logstash 位，我不需要那個。 然后將以下內容添加到.bashrc

export ELASTICSEARCH_HOST=$(hostname)
export ELASTICSEARCH_PASSWORD=foobar
export ELASTICSEARCH_USERNAME=elastic
export ELASTIC_VERSION=7.4.2
export INITIAL_MASTER_NODES=$ELASTICSEARCH_HOST

並且docker stack deploy --compose-file docker-compose.yml elastic工作。

Answer 5

我從 Ahmet Vehbi Olgaç 的 docker-compose.yml 中收集到的想法對我有用：

1. 使用deployment / mode: global 。 對於像這樣配置的每個節點，這將導致 swarm 為每個 swarm worker 部署一個副本。

2. 使用deployment / endpoint_mode: dnsrr 。 這將使 swarm 中的所有容器通過服務名稱訪問節點。

3. 使用hostname: {{.Node.Hostname}}或類似的基於模板的表達式。 這確保了每個已部署容器的唯一名稱。

4. 使用environment / node.name={{.Node.Hostname}} 。 同樣，您可以改變模式。 關鍵是每個 es 節點都應該有一個唯一的名字。

5. 使用cluster.initial_master_nodes=*hostname1*,*hostname2*,... 。 假設您知道 docker worker 機器的主機名。 使用您在#3 中使用的任何模式，但替換掉整個主機名，並包括所有主機名。
   
   如果您不知道您的主機名，您可以按照 Andrew Cachia 的回答建議進行操作：設置一個容器（不要復制它）僅充當主種子並為其提供一個可預測的主機名，然后讓所有其他節點引用該容器節點作為主種子。 但是，這會引入單點故障。

Answer 6

Elasticsearch 8.5.0 答案。

根據我的需要，我不想添加反向代理/負載均衡器，但我確實想在運行 Elasticsearch 副本的群節點上公開端口 9200（僅使用群），以便外部客戶端可以訪問 Elasticsearch休息API。 因此，我使用端點模式dnsrr ( ref ) 並在運行副本的主機上公開端口 9200。

如果您不需要公開端口 9200（即，沒有任何東西會連接到 swarm 外部的elasticsearch副本），請從elasticsearch服務中刪除ports: config。

我也只希望 elasticsearch 副本在我的群節點的一個子集（其中 3 個）上運行。 我在這三個節點上創建了elasticsearch node label elasticsearch。 然后mode: global和約束node.labels.elasticsearch==True將確保 1 個副本在每個節點上運行。

我也在這 3 個節點之一上運行 kibana：swarm 可以選擇哪一個，因為端口 5601 暴露在 swarm 的入口覆蓋網絡上。

您可能需要編輯的行由###### 。

# docker network create -d overlay --attachable elastic-net

# cat elastic-stack-env
#!/bin/bash
export STACK_VERSION=8.5.0   # Elasticsearch and Kibana version
export ES_PORT=9200          # port to expose Elasticsearch HTTP API to the host
export KIBANA_PORT=5601      # port to expose Kibana to the host

read -p "Enter elastic user password: " ELASTIC_PASSWORD
read -p "Enter kibana_system user password: " KIBANA_PASSWORD
export KIBANA_URL=https://kibana.my-domain.com:$KIBANA_PORT   #######
export SHARED_DIR=/some/nfs/or/shared/storage/elastic         #######
export KIBANA_SSL_KEY_PATH=config/certs/kibana.key
export KIBANA_SSL_CERT_PATH=config/certs/kibana.crt
export ELASTIC_NODES=swarm_node1,swarm_node2,swarm_node3      #######
  # ELASTIC_NODES must match what docker reports from {{.Node.Hostname}}
export KIBANA_SSL_CERT_AUTH_PATH=config/certs/My_Root_CA.crt  #######
export CLUSTER_NAME=docker-cluster
export MEM_LIMIT=4294967296  # 4 GB; increase or decrease based on the available host memory (in bytes)

# cat elastic-stack.yml
version: "3.8"
services:
  elasticsearch:
    image: localhost:5000/elasticsearch:${STACK_VERSION:?}  ####### I have a local registry
    deploy:
      endpoint_mode: dnsrr
      mode: global  # but note constraints below
      placement:
        constraints:
          - node.labels.elasticsearch==True
      resources:
        limits:
          memory:
            ${MEM_LIMIT}
    dns: 127.0.0.11  # use docker DNS only (may not be required)
    networks:
      - elastic-net
    volumes:
      - ${SHARED_DIR:?}/certs:/usr/share/elasticsearch/config/certs
      - /path/to/some/local/storage/elasticsearch:/usr/share/elasticsearch/data
    ports:  ##### remove if nothing outside of swarm needs to access port 9200
      - target: 9200
        published: ${ES_PORT}  # we publish this port so that external clients can access the ES REST API
        protocol: tcp
        mode: host  # required when using dnsrr
    environment:    # https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
                    # https://www.elastic.co/guide/en/elasticsearch/reference/master/docker.html#docker-configuration-methods
      - node.name={{.Node.Hostname}}  # see Andrew Cachia's answer
      - cluster.name=${CLUSTER_NAME}
      - discovery.seed_hosts=elasticsearch             # use service name here, since (docker's) DNS is used:
        # https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#unicast.hosts
      - cluster.initial_master_nodes=${ELASTIC_NODES}  # use node.names here
        # https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#initial_master_nodes
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
    healthcheck:
      test:
        [ "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
    logging:        # we use rsyslog
      driver: syslog
      options:
        syslog-facility: "local2"

  kibana:
    # this service depends on the setup service (defined below), but docker stack has no
    # way to specify dependencies, but more importantly, there's been a move away from this:
    # https://stackoverflow.com/a/47714157/215945
    image: localhost:5000/kibana:${STACK_VERSION:?}    ######
    hostname: kibana
    deploy:
      placement:
        constraints:
          - node.labels.elasticsearch==True  # run KB on any one of the ES nodes
      resources:
         limits:
          memory:
            ${MEM_LIMIT}
    dns: 127.0.0.11  # use docker DNS only (may not be required)
    networks:
      - elastic-net
    volumes:
      - ${SHARED_DIR:?}/kibana:/usr/share/kibana/data
      - ${SHARED_DIR:?}/certs:/usr/share/kibana/config/certs
    ports:
      - ${KIBANA_PORT}:5601
    environment:  # https://www.elastic.co/guide/en/kibana/master/settings.html
                  # https://www.elastic.co/guide/en/kibana/master/docker.html#environment-variable-config
                  # CAPS_WITH_UNDERSCORES must be used with Kibana
      - SERVER_NAME=kibana
      - ELASTICSEARCH_HOSTS=["https://elasticsearch:9200"]
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - SERVER_PUBLICBASEURL=${KIBANA_URL}
      # if you don't want to use https/TLS with Kibana, comment-out
      # the next four lines
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY=${KIBANA_SSL_KEY_PATH}
      - SERVER_SSL_CERTIFICATE=${KIBANA_SSL_CERT_PATH}
      - SERVER_SSL_CERTIFICATEAUTHORITIES=${KIBANA_SSL_CERT_AUTH_PATH}
      - TELEMETRY_OPTIN=false
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -sIk https://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
    logging:
      driver: syslog
      options:
        syslog-facility: "local2"

  setup:
    image: localhost:5000/elasticsearch:${STACK_VERSION:?}   #######
    deploy:
      placement:
        constraints:
          - node.labels.elasticsearch==True
      restart_policy:  # https://docs.docker.com/compose/compose-file/compose-file-v3/#restart_policy
        condition: none
    volumes:
      - ${SHARED_DIR:?}/certs:/usr/share/elasticsearch/config/certs
    dns: 127.0.0.11  # use docker DNS only (may not be required)
    networks:
      - elastic-net
    command: >
      bash -c '
        until curl -s --cacert config/certs/ca/ca.crt https://elasticsearch:9200 | grep -q "missing authentication credentials"
        do
          echo "waiting 30 secs for Elasticsearch availability..."
          sleep 30
        done
        echo "setting kibana_system password"
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://elasticsearch:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"
        do
          echo "waiting 10 secs before trying to set password again..."
          sleep 10
        done
        echo "done"
      '
    logging:
      driver: syslog
      options:
        syslog-facility: "local2"

networks:
  elastic-net:
    external: true

部署：

# . ./elastic-stack-env
# docker stack deploy -c elastic-stack.yml elastic
# # ... after Kibana comes up, you can remove the setup service if you want:
# docker service rm elastic_setup

以下是我創建 Elasticsearch CA 和證書的方式：

# cat elastic-certs.yml
version: "3.8"
services:
  setup:
    image: localhost:5000/elasticsearch:${STACK_VERSION:?}   #######
    volumes:
      - ${SHARED_DIR:?}/certs:/usr/share/elasticsearch/config/certs
    user: "0:0"
    command: >
      bash -c '
        if [ ! -f certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: elasticsearch\n"\
          "    dns:\n"\
          "      - elasticsearch\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
          echo "Setting file permissions"
          chown -R root:root config/certs;
          find . -type d -exec chmod 750 \{\} \;;
          find . -type f -exec chmod 640 \{\} \;;
        fi;
        sleep infinity
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/elasticsearch/elasticsearch.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

# . ./elastic-stack-env
# docker stack deploy -c elastic-certs.yml elastic-certs
# # ... ensure files are created under $SHARED_DIR/certs, then
# docker stack rm elastic-certs

我如何創建 Kibana 證書超出了這個問題的范圍。

我運行 Fluent Bit swarm 服務（ mode: global ，docker network elastic-net ）將日志發送到 elasticsearch 服務。 雖然超出了這個問題的范圍，但這是重要的配置：

[OUTPUT]
    name          es
    match         <whatever is appropriate for you here>
    host          elasticsearch
    port          9200
    index         my-index-default
    http_user     fluentbit
    http_passwd   ${FLUENTBIT_PASSWORD}
    tls           on
    tls.ca_file   /certs/ca/ca.crt
    tls.crt_file  /certs/elasticsearch/elasticsearch.crt
    tls.key_file  /certs/elasticsearch/elasticsearch.key
    retry_limit   false
    suppress_type_name on
#   trace_output  on

Host elasticsearch會被docker的DNS服務器解析到elasticsearch副本的三個IP地址，所以不存在單點故障。