如何从现有日志中提取字段（ECS 中的fluent-bit）

Question

I have configured Fluent-bit on my ECS cluster. I can see the logs in Kibana. But all the log data are sent to a single field "log". How can I extract each field into a separate field. There is a solution for fluentd already in this question.

But how can I achieve the same with fluent-bit?

There is a solution in Kuberntetes with fluent-bit: https://docs.fluentbit.io/manual/filter/kubernetes

How do I achieve the same thing in ECS?

Answer 1

Generally fluent-bit send exactly docker log file that taking from /var/lib/docker/containers/*/*.log You can browse this path on your machine and see that it contains JSON strings with exactly two fields you mentioned.

From here you have number ways, I'll discover two that I know well:

1. Use logstash :

   You should know well the log structure. This helps you to create the right filters pipeline for the parse log field. Usually, people use filter plugins for this. If you add log examples I will be able to make an example of a filter like this

2. Use the elasticsearch ingest node .

   You should know well the log structure. For be able easy to create processors pipeline for parse log field. More one time, specific log examples help's us to help you.

The most used filter/processor is grok filter / processor . This tool have a lot of options for parse structured text from any log.