elasticsearch on kube with persistent volume

Question

I was trying elasticsearch deployment on Kubernetes (EKS) cluster with persistence volume (EBS), refering to https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: elasticsearch
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          imagePullPolicy: IfNotPresent
          command: ["sysctl", "-w", "vm.max_map_count=262144"]
          securityContext:
            privileged: true
      containers:
      - name: elasticsearch
        image: elasticsearch:6.6.1
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
          capabilities:
            add:
              - IPC_LOCK
        ports:
        - containerPort: 9300
          protocol: TCP
        resources:
          limits:
            cpu: 1
        env:
        - name: CLUSTER_NAME
          value: elasticsearch
        - name: ES_JAVA_OPTS
          value: "-Xms256m -Xmx512m"
        volumeMounts:
          - name: elasticsearch-storage
            mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: elasticsearch-storage
        persistentVolumeClaim:
          claimName: elasticsearch

But pod failed with

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-03-13T18:03:08,459][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: Failed to create node environment
    at org.elasticsearch.node.Node.<init>(Node.java:298) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
    ... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?]
    at java.nio.file.Files.createDirectory(Files.java:689) ~[?:?]
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:796) ~[?:?]
    at java.nio.file.Files.createDirectories(Files.java:782) ~[?:?]
    at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
    ... 6 more

From this discussion , it looks like permission issue on /usr/share/elasticsearch/data dir.

following initContainers helped to get me around this issue.

initContainers:
  - name: init-sysctl
    image: busybox
    imagePullPolicy: IfNotPresent
    #command: ["sysctl", "-w", "vm.max_map_count=262144"]
    command: ["/bin/sh","-c"]
    args: ["sysctl -w vm.max_map_count=262144; chown -R 1000:1000 /usr/share/elasticsearch/data"]
    securityContext:
      privileged: true
    volumeMounts:
      - name: elasticsearch-storage
        mountPath: /usr/share/elasticsearch/data

Whats is correct way to fix this issue ?

Answer 1

The solution you are using is a workaround , but you can do it a bit kubernetes way is explained here:

Permission denied when chown on elasticsearch data directory in kubernetes statefulset

So in your case it should be like:

....

spec:
  ...
  securityContext:
    fsGroup: 1000

Answer 2

I was trying elasticsearch deployment on Kubernetes (EKS) cluster with persistence volume (EBS), refering to https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: elasticsearch
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          imagePullPolicy: IfNotPresent
          command: ["sysctl", "-w", "vm.max_map_count=262144"]
          securityContext:
            privileged: true
      containers:
      - name: elasticsearch
        image: elasticsearch:6.6.1
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
          capabilities:
            add:
              - IPC_LOCK
        ports:
        - containerPort: 9300
          protocol: TCP
        resources:
          limits:
            cpu: 1
        env:
        - name: CLUSTER_NAME
          value: elasticsearch
        - name: ES_JAVA_OPTS
          value: "-Xms256m -Xmx512m"
        volumeMounts:
          - name: elasticsearch-storage
            mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: elasticsearch-storage
        persistentVolumeClaim:
          claimName: elasticsearch

But pod failed with

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-03-13T18:03:08,459][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: Failed to create node environment
    at org.elasticsearch.node.Node.<init>(Node.java:298) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
    ... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?]
    at java.nio.file.Files.createDirectory(Files.java:689) ~[?:?]
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:796) ~[?:?]
    at java.nio.file.Files.createDirectories(Files.java:782) ~[?:?]
    at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
    ... 6 more

From this discussion , it looks like permission issue on /usr/share/elasticsearch/data dir.

following initContainers helped to get me around this issue.

initContainers:
  - name: init-sysctl
    image: busybox
    imagePullPolicy: IfNotPresent
    #command: ["sysctl", "-w", "vm.max_map_count=262144"]
    command: ["/bin/sh","-c"]
    args: ["sysctl -w vm.max_map_count=262144; chown -R 1000:1000 /usr/share/elasticsearch/data"]
    securityContext:
      privileged: true
    volumeMounts:
      - name: elasticsearch-storage
        mountPath: /usr/share/elasticsearch/data

Whats is correct way to fix this issue ?