Kubernetes Persistent Volume is not working on GCE

Question

I am trying to make my elastic search pods persistent so that data is preserved when deployment or pods are recreated.Elastic search is a part of Graylog2 setup.

After I set everything up, I sent a few logs to Graylog and I could see them appear on the dashboard. However, I deleted elasticsearch pod and after it was recreated all the data was lost on Graylog dashboard.

I am using GCE.

Here is my persistent volume config:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: elastic-pv
  labels:
    type: gcePD
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteOnce
  gcePersistentDisk:
    fsType: ext4
    pdName: elastic-pv-disk

Persistent volume claim config:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: elastic-pvc
  labels:
    type: gcePD
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi

and here is my elasticsearch deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elastic-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        type: elasticsearch
    spec:
      containers:
      - name: elastic-container
        image: gcr.io/project/myelasticsearch:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 9300
          name: first-port
          protocol: TCP
        - containerPort: 9200
          name: second-port
          protocol: TCP
        volumeMounts:
            - name: elastic-pd
              mountPath: /data/db
      volumes:
      - name: elastic-pd
        persistentVolumeClaim:
          claimName: elastic-pvc

Output of kubectl describe pod :

Name:       elastic-deployment-1423685295-jt6x5
Namespace:  default
Node:       gke-sd-logger-default-pool-2b3affc0-299k/10.128.0.6
Start Time: Tue, 09 May 2017 22:59:59 +0500
Labels:     pod-template-hash=1423685295
        type=elasticsearch
Status:     Running
IP:     10.12.0.11
Controllers:    ReplicaSet/elastic-deployment-1423685295
Containers:
  elastic-container:
    Container ID:   docker://8774c747e2a56363f657a583bf5c2234ed2cff64dc21b6319fc53fdc5c1a6b2b
    Image:      gcr.io/thematic-flash-786/myelasticsearch:v1
    Image ID:       docker://sha256:7c25be62dbad39c07c413888e275ae419a66070d37e0d98bf5008e15d7720eec
    Ports:      9300/TCP, 9200/TCP
    Requests:
      cpu:      100m
    State:      Running
      Started:      Tue, 09 May 2017 23:02:11 +0500
    Ready:      True
    Restart Count:  0
    Volume Mounts:
      /data/db from elastic-pd (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qtdbb (ro)
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True
  Ready     True
  PodScheduled  True
Volumes:
  elastic-pd:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elastic-pvc
    ReadOnly:   false
  default-token-qtdbb:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-qtdbb
QoS Class:  Burstable
Tolerations:    <none>
No events.

Output of kubectl describe pv :

Name:       elastic-pv
Labels:     type=gcePD
StorageClass:
Status:     Bound
Claim:      default/elastic-pvc
Reclaim Policy: Retain
Access Modes:   RWO
Capacity:   200Gi
Message:
Source:
    Type:   GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName: elastic-pv-disk
    FSType: ext4
    Partition:  0
    ReadOnly:   false
No events.

Output of kubectl describe pvc :

Name:       elastic-pvc
Namespace:  default
StorageClass:
Status:     Bound
Volume:     elastic-pv
Labels:     type=gcePD
Capacity:   200Gi
Access Modes:   RWO
No events.

Confirmation that real disk exists:

What could be the reason Persistent Volume is not persistent?

Answer 1

In the official images, the Elasticsearch data is stored at /usr/share/elasticsearch/data and not /data/db . It would appear that you needed to updated the mount to be /usr/share/elasticsearch/data instead to get the data storing on the persistent volume.