Fluentd time format in parser section doesn't work

Question

Hi Fluentd Experts and Users!

I found that Fluentd parser cannot parse my time field in json format.

Frankly, I don't know what the time should look like after going through the parser, because I've never had success with it. The configuration I refer to is this link: https://docs.fluentd.org/configuration/parse-section#time-parameters

My test case is simple, input is a dummy Json with time field {"hello":"world", "time":"1991-02-19 00:00:00"} .

But no matter how I modify the time format, or deliberately write the wrong format of the input time, it will not affect the final output, and the final output seems to just print the time field as a string.

Here is my fluentd config:

<source>
  @type dummy
  dummy {"hello":"world", "time":"1991-02-19 00:00:00"}
  tag sample
  <parse>
    @type json
    time_type string
    time_format "%Y-%m-%dT%H:%M:%S"
    time_key "time"
    keep_time_key true
  </parse>
</source>

<match sample>
  @type stdout
  @id out_stdout
</match>

and the output is:

2022-04-12 01:44:01.034400852 +0000 sample: {"hello":"world","time":"1991-02-19 00:00:00"}
2022-04-12 01:44:02.035744518 +0000 sample: {"hello":"world","time":"1991-02-19 00:00:00"}
2022-04-12 01:44:03.037178309 +0000 sample: {"hello":"world","time":"1991-02-19 00:00:00"}
2022-04-12 01:44:04.038867836 +0000 sample: {"hello":"world","time":"1991-02-19 00:00:00"}
....

You can see that although there missed a 'T' in the input time field, but there is no error or warning, and the output just send the raw string out.

And if I put the 'T' back to the input, the output will get the 'T' back too. The config and output are:

<source>
  @type dummy
  dummy {"hello":"world", "time":"1991-02-19T00:00:00"}
  tag sample
  <parse>
    @type json
    time_type string
    time_format "%Y-%m-%dT%H:%M:%S"
    time_key "time"
    keep_time_key true
  </parse>
</source>

<match sample>
  @type stdout
  @id out_stdout
</match>

2022-04-12 01:59:56.026574094 +0000 sample: {"hello":"world","time":"1991-02-19T00:00:00"}
2022-04-12 01:59:57.028190722 +0000 sample: {"hello":"world","time":"1991-02-19T00:00:00"}
2022-04-12 01:59:58.029744904 +0000 sample: {"hello":"world","time":"1991-02-19T00:00:00"}
2022-04-12 01:59:59.031296625 +0000 sample: {"hello":"world","time":"1991-02-19T00:00:00"}
...

So I am very confused, does the time format in fluentd's parser really work? And I'm not sure if this is a Fluentd issue, or Plugin issue, or my configuration is wrong.

Thanks a lot in advance!

Answer 1

The dummy / sample input plugin doesn't support a <parse> section.

Just ran fluentd with your configuration and observed these warnings in the logs:

2022-04-12 19:44:56 +0500 [warn]: section <parse> is not used in <source> of sample plugin
2022-04-12 19:44:56 +0500 [warn]: section <parse> is not used in <source> of sample plugin
2022-04-12 19:44:56 +0500 [warn]: section <parse> is not used in <source> of sample plugin
2022-04-12 19:44:56 +0500 [warn]: section <parse> is not used in <source> of sample plugin
2022-04-12 19:44:56 +0500 [warn]: section <parse> is not used in <source> of sample plugin

Instead, you might want to add the <filter> section with type parser configured for json format.

You should always check the logs for any issues. In addition, you can turn on debug logs with -v flag or trace logs with -vv flag.

The --dry-run flag to pretty handly to validate the configuration file eg:

fluentd -c fluent.conf --dry-run

fluentd --help will give you the complete list of available flags.