Fluentd sidecar isn't able to send logs to Elasticsearch: types removal Specifying types in bulk requests is deprecated

Question

I have a sidecar container configured to collect logs for my app container.

I first tested my fleunt.conf using the following configuration:

<source>
    @type forward
    bind "127.0.0.1"
    port 24224
    <parse>
      @type json
    </parse>
  </source>
  <match app.default>
    @type stdout
  </match>

It worked fine when I tailed the log of fluentd container in my pod using kubectl , I could see my app logs in JSON format.

Now, I am trying to send the logs to elasticsearch . Here is my fluent.conf for the same:

<source>
    @type forward
    bind "127.0.0.1"
    port 24224
    <parse>
      @type json
    </parse>
  </source>
  <match app.default>
    @type elasticsearch
    host "elasticsearch_host"
    port 9200
    index_name "app-log"
    user "log_user"
    password xxxxxx
  </match>

The fluentd container is running fine inside my pod but I am not able to define the index that I have configured in fluetn.conf in Kibana. Looks like the index pattern is not getting loaded into elasticsearch .

The fluentd container is showing the following warning:

2020-10-22 12:31:10 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-10-22 12:31:10 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.0'
2020-10-22 12:31:10 +0000 [info]: gem 'fluentd' version '1.10.4'
2020-10-22 12:31:12 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    bind "127.0.0.1"
    port 24224
    <parse>
      @type json
    </parse>
  </source>
  <match app.default>
    @type elasticsearch
    host "elasticsearch_host"
    port 9200
    index_name "app-log"
    user "log_user"
    password xxxxxx
  </match>
</ROOT>
2020-10-22 12:31:12 +0000 [info]: starting fluentd-1.10.4 pid=8 ruby="2.5.8"
2020-10-22 12:31:12 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2020-10-22 12:31:16 +0000 [info]: adding match pattern="app.default" type="elasticsearch"
2020-10-22 12:31:17 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-10-22 12:31:17 +0000 [info]: adding source type="forward"
2020-10-22 12:31:17 +0000 [warn]: section <parse> is not used in <source> of forward plugin
2020-10-22 12:31:17 +0000 [info]: #0 starting fluentd worker pid=22 ppid=8 worker=0
2020-10-22 12:31:17 +0000 [info]: #0 listening port port=24224 bind="127.0.0.1"
2020-10-22 12:31:17 +0000 [info]: #0 fluentd worker is now running worker=0
warning: 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."
warning: 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."
warning: 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."

How do I get rid of this warning "[types removal] Specifying types in bulk requests is deprecated." ? Is there any additional configuration to the fluent.conf that can help me get past this?

I tried using suppress_type_name true in the configuration but it did not work.

Answer 1

Support for suppress_type_name true was only added with version 4.0.10 of the plugin: https://github.com/uken/fluent-plugin-elasticsearch/blob/master/History.md#4010 .

According to the logs, you're running 4.0.0, so upgrading and setting the flag in the config should get rid of the warning. The warning should not affect the bulk request.

Answer 2

I got the same depreciation warning even with fluent-plugin-elasticsearch-5.0.3

starting fluentd-1.12.4 pid=7 ruby="2.6.7"
2022-01-10 13:25:43 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "/fluentd/etc/../../etc/fluent/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "-r", "/fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]

warning: 299 Elasticsearch-7.12.xxxxxxxxx "[types removal] Specifying types in bulk requests is deprecated."
2022-01-10 13:32:09 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 17, pod_cache_size: 241, namespace_cache_miss: 88, pod_cache_api_updates: 5, id_cache_miss: 5, pod_cache_watch_delete_ignored: 2, pod_cache_watch_updates: 15, pod_cache_watch_misses: 8, pod_cache_watch_ignored: 2, pod_cache_host_updates: 241, namespace_cache_host_updates: 17