Fluentd capture when a Kubernetes Pod terminates with the 'CrashLoopBackOff'?

Question

I'm running a pod that write a simple message to the 'terminationMessagePath' then the pod exit with "CrashLoopBackOff". I would like to be able to debug through Kibana instead of having to login to each Kubernetes nodes. I queried Kibana to get the container last state value "CrashLoopBackOff" from the property reason & message and could not locate an entry.

I can see the fields for the pod in Kibana but the field that I'm looking for (in bold yaml format below) is empty.

What configuration is needed in fluentd to get the log from Kubernetes pod? or configuration need to be set from Kubernetes

$ kubectl get pod_name_1 -o=yaml

terminationMessagePath: /var/log/containers/dt.log
volumeMounts:
- mountPath: /var/log/containers
  name: data
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: default-token-s0w2n
  readOnly: true
dnsPolicy: ClusterFirst
nodeName: dev-master-01
restartPolicy: Always
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
  path: /var/log/containers
  name: data
- name: default-token-s0w2n
secret:
  defaultMode: 420
  secretName: default-token-s0w2n
status:
conditions:
 - lastProbeTime: null
   lastTransitionTime: 2017-07-05T14:45:11Z
   status: "True"
   type: Initialized
 - lastProbeTime: null
   lastTransitionTime: 2017-07-05T17:00:22Z
   message: 'containers with unready status: [dt-termination-demo]'
   reason: ContainersNotReady
   status: "False"
   type: Ready
 - lastProbeTime: null
   lastTransitionTime: 2017-07-05T14:45:11Z
   status: "True"
   type: PodScheduled
   containerStatuses:
 - containerID: 
   docker://9649c26527cf0e1cd3bd67ba9c606c0b78e6b4f08bacf96175627ddc7d250772
   image: debian
   imageID: docker pullable://docker.io/debian@sha256:
            7d067f77d2ae5a23fe6920f8fbc2936c4b0d417e9d01b26372561860750815f0
   lastState:
   terminated:
   containerID: docker://
   9649c26527cf0e1cd3bd67ba9c606c0b78e6b4f08bacf96175627ddc7d250772              
   exitCode: 0
   finishedAt: 2017-07-05T17:00:22Z
   **message: |
    Sleep expired**
    reason: Completed
   startedAt: 2017-07-05T17:00:12Z
   name: dt-termination-demo
   ready: false
   restartCount: 30
   state:
     waiting:
     message: Back-off 5m0s restarting failed container=dt-termination-demo 
              pod=dt-termination-demo-2814930607-8kshj_
              default(8c247b15-6190-11e7-acb7-00505691210d)
     **reason: CrashLoopBackOff**
   hostIP: 192.21.19.128
   phase: Running
   podIP: 10.0.0.8
   startTime: 2017-07-05T14:45:11Z

Answer 1

When Fluentd is deployed as a DaemonSet, it aims to collect all logs from the Node and Pods. As a guide to accomplish this please check the following Yaml file and further repository associated:

• https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml
• https://github.com/fluent/fluentd-kubernetes-daemonset

If you need additional assistance you can also join our Slack channel:

• http://slack.fluentd.org