GKE 集群節點以 CrashLoopBackOff 結束

Question

我在 GKE 中有一個 3 節點設置。 我的 pod 創建之一是在 CrashLoopBackOff state 中，它沒有恢復。 日志建議以下 java.lang.IllegalArgumentException。 但其他 2 個吊艙沒有這樣的問題。 他們已經啟動並運行。 我完全不確定這個問題，有人可以幫助我嗎？

問題是 YML 文件中install-plugins的副產品嗎？ 如果是，為什么其他 pod 沒有出現同樣的問題？ 你能幫我嗎？

例外：

    "type": "server", "timestamp": "2022-08-29T19:52:29,743Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "dev", "node.name": "dev-es-data-hot-1", "message": "uncaught exception in thread [main]", 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]",
"Caused by: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:561) ~[elasticsearch-7.16.3.jar:7.16.3]",
uncaught exception in thread [main]
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:507) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:477) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:137) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.node.Node.<init>(Node.java:500) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]",
"... 6 more"] }

這是我的 YAML 配置：

 - name: data-hot-ingest
   count: 3
   config:
     node.roles: ["data_hot", "ingest", "data_content"]
     node.attr.data: hot
     node.store.allow_mmap: false
     xpack.security.authc:
       anonymous:
         username: anon
         roles: monitoring_user
   podTemplate:
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: type
                 operator: In
                 values:
                 - hot
       initContainers:
       - name: install-plugins
         command:
         - sh
         - -c
         - |
           bin/elasticsearch-plugin install --batch repository-gcs
       - name: set-virtual-mem
         command:
         - sysctl
         - -w
         - vm.max_map_count=262144
       containers:
         - name: elasticsearch
           resources:
             requests:
               memory: "64Gi"
               cpu: "30000m"
             limits:
              memory: "65Gi"
              cpu: "30000m"
           env:
             - name: ES_JAVA_OPTS
               value: -Xms32g -Xmx32g
           readinessProbe:
             httpGet:
               scheme: HTTPS
               port: 8080
   volumeClaimTemplates:
   - metadata:
       name: elasticsearch-data
     spec:
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: 350Gi
       storageClassName: gold

編輯：我們配置了這個安全設置，它鏈接到我們的一個秘密

  secureSettings:
  - secretName: credentials

Answer 1

[回答我自己的問題]

嘗試解決以下異常： java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json]

我嘗試比較 Pod 的 yaml 配置，發現成功運行的 Pod 沒有安全設置。 但是崩潰循環的 pod 在elastic-internal-secure-settings下具有安全設置

- name: elastic-internal-secure-settings
    secret:
      defaultMode: 420
      optional: false
      secretName: dev-es-secure-settings

在運算符 yaml 中，我發現了這個：

secureSettings:
  - secretName: credentials

只是為了確認行為，我升級了 statefulset，發現新的 pod 也崩潰循環並出現相同的錯誤。 因此，上個月有人嘗試了安全設置，但它使 pod 崩潰循環，並且沒有將其重置為正常狀態。 一旦我從操作員 yaml 中刪除了安全設置，吊艙就開始運行而沒有任何問題。