如果 Loki pod 在 EKS Fargate 中意外终止，EFS 是否是一个好的日志备份选项

Question

I am currently using Loki to store logs generated by my applications from EKS Fargate. Sidecar pattern with promtail is used to scrape logs. Single Loki pod is used and S3 is configured as a destination to store logs. It works nicely as expected. However, when I tested the availability of the logging system by deleting pods, I discovered that if Loki's pod was deleted, some logs would be missing (range around 20 mins before the pod was deleted to the time the pod was deleted) even after the pod restarted.

To solve this problem, I tried to use EFS as the persistent volume of Loki' pod, mounting the path /loki. The whole process is followed by this article ( https://aws.amazon.com/blogs/aws/new-aws-fargate-for-amazon-eks-now-supports-amazon-efs/ ). But I have got an error from the Loki pod with msg "error running loki" err="mkdir /loki/compactor: permission denied”

Therefore, I have 2 questions in my mind:

Should I use EFS as a solution for log backup in my case? Why did I get a permission denied inside the pod, any ways to solve this problem?

My Loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  # grpc_listen_port: 9096

ingester:
  wal:
    enabled: true
    dir: /loki/wal
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    # final_sleep: 0s
  chunk_idle_period: 3m       
  chunk_retain_period: 30s    
  max_transfer_retries: 0     
  chunk_target_size: 1048576 

schema_config:
  configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

  aws:
    bucketnames: bucketnames
    endpoint: s3.us-west-2.amazonaws.com
    region: us-west-2
    access_key_id: access_key_id
    secret_access_key:  secret_access_key
    sse_encryption: true

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  compaction_interval: 5m

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 48h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 96h

querier:
  query_ingesters_within: 0

analytics:
  reporting_enabled: false

Deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: fargate-api-dev
  name: dev-loki
spec:
  selector:
    matchLabels:
      app: dev-loki
  template:
    metadata:
      labels:
        app: dev-loki
    spec:
      volumes:
        - name: loki-config
          configMap:
            name: dev-loki-config
        - name: dev-loki-efs-pv
          persistentVolumeClaim:
            claimName: dev-loki-efs-pvc


      containers:
        - name: loki
          image: loki:2.6.1
          args:
            - -print-config-stderr=true
            - -config.file=/tmp/loki.yaml
         
          resources:
            limits:
              memory: "500Mi"
              cpu: "200m"
          ports:
            - containerPort: 3100
          volumeMounts:
            - name: dev-loki-config
              mountPath: /tmp
              readOnly: false
             - name: dev-loki-efs-pv
               mountPath: /loki

Promtail-config.yaml

server:
  log_level: info
  http_listen_port: 9080

clients:
  - url: http://loki.com/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  - job_name: api-log
    static_configs:
    - targets:
      - localhost
      labels:
        job: apilogs
        pod: ${POD_NAME}
        __path__: /var/log/*.log  

Answer 1

I had a similar issue using EFS as volume to store the logs and I found this solution https://github.com/grafana/loki/issues/2018#issuecomment-1030221498

Basically loki container by it's own is not able to create a directory to start working, so we used a initcotainer to do it for it.

This solution worked like a charm for.