Fluentd elasticsearch plugin not connecting to Elasticsearch from Kubernetes on a Raspberry Pi

Question

EDIT: I've added new information per my comment in response to efrat-levitan's suggestion. The log output listed is slightly different as I upgraded my Elasticsearch version to 7.6.0 as suggested by the original comment. To help debugging, I also didn't start Elasticsearch right away. The effect of this can be seen in the ECONNREFUSED messages in the log. I called out the log file changes in the summary below. Most of the rest of the message text (ie, not log snippets) remains the same as before.

I've been working on getting an ARM version (for a Raspberry Pi 3 & 4) of fluentd with the fluent-plugin-elasticsearch plugin running in docker. I haven't been able to find an appropriate docker image so I've built one on my own (if anyone knows where I can find one I'd appreciate it). I started with the fluentd-docker-image repo (doesn't include Elasticsearch plugins) and modified it as I thought necessary using the fluentd-kubernetes-daemonset repo (does include the Elasticsearch plugins). The good news is that it starts up just fine on a Raspberry Pi. The bad news is that it appears like it doesn't even try to connect to ElasticSearch (external to the Raspberry Pi network. The log file looks like this:

2020-03-02 18:13:15 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.12'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.4'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluentd' version '1.9.2'
2020-03-02 18:13:16 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:16 +0000 [info]: using configuration file: <ROOT>
  <filter **>
    @type stdout
  </filter>
  <source>
    @type forward
    @label @ES
  </source>
  <label @ES>
    <match out.elasticsearch.**>
      @type elasticsearch
      @log_level "info"
      include_tag_key true
      host "10.0.0.223"
      port 9200
      path ""
      scheme http
      index_name "logstash"
      include_timestamp true
      log_es_400_reason false
      logstash_prefix "logstash"
      logstash_dateformat "%Y.%m.%d"
      logstash_format true
      ssl_verify true
      ssl_version TLSv1_2
      user
      password xxxxxx
      reload_connections false
      reconnect_on_error true
      reload_on_failure true
      request_timeout 5s
      sniffer_class_name "Fluent::Plugin::ElasticsearchSimpleSniffer"
      type_name "doc"
      template_name
      template_file
      template_overwrite false
      time_key "@timestamp"
      <buffer>
        flush_thread_count 8
        flush_interval 5s
        chunk_limit_size 2M
        queue_limit_length 32
        retry_max_interval 30
        retry_forever true
      </buffer>
    </match>
  </label>
  <label @ERROR>
    <match **>
      @type stdout
    </match>
  </label>
</ROOT>
2020-03-02 18:13:16 +0000 [info]: starting fluentd-1.9.2 pid=7 ruby="2.6.5"
2020-03-02 18:13:16 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "-r", "/usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:23 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2020-03-02 18:13:27 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:27 +0000 [warn]: #0 Remaining retry: 13. Retry to communicate after 4 second(s).
2020-03-02 18:13:35 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:35 +0000 [warn]: #0 Remaining retry: 12. Retry to communicate after 8 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:51 +0000 [warn]: #0 Remaining retry: 11. Retry to communicate after 16 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-03-02 18:13:51 +0000 [info]: adding match in @ERROR pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding filter pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding source type="forward"
2020-03-02 18:13:51 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:51 +0000 [info]: #0 starting fluentd worker pid=22 ppid=7 worker=0
2020-03-02 18:13:51 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2020-03-02 18:13:51 +0000 [info]: #0 fluentd worker is now running worker=0
2020-03-02 18:13:51.581170450 +0000 fluent.info: {"pid":22,"ppid":7,"worker":0,"message":"starting fluentd worker pid=22 ppid=7 worker=0"}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.585716902 +0000 fluent.info: {"port":24224,"bind":"0.0.0.0","message":"listening port port=24224 bind=\"0.0.0.0\""}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.593737828 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}

To narrow in on the log a little more, fluentd does seem to know about Elasticsearch, both in config and in connectivity:

2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)

The 'ECONNREFUSED' error is because I stopped Elasticsearch to help out with troubleshooting. So Fluentd is trying to connect. As shown in the next snippet, after starting Elasticsearch, it does connect and continue processing:

2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.

The problem is that Fluentd doesn't appear to actually complete the "sign" process with Elasticsearch. I would expect to see something like this on success, or some sort of error message.

2020-02-28 21:56:26 +0000 [info]: #0 [out_es] Connection opened to Elasticsearch cluster => {:host=>"10.0.0.223", :port=>9200, :scheme=>"http", :path=>""}

I'd also expect to see some evidence in Elasticsearch that it did indeed complete the "signon" process. Eg, I don't see a "logstash" index in Elasticsearch, nor do I see any evidence in the Elasticsearch log that any client other than Kibana is connected.

The configuration is logged above and seems to be correct to me. The command line recorded in the log looks fine to me too.

/usr/local/bin/ruby -Eascii-8bit:ascii-8bit /usr/local/bundle/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -r /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb --under-supervisor

Logging into the pod and connecting to Elasticsearch works as well:

$ ks exec -it fluentd-h2qzn sh
$ curl http://10.0.0.223:9200
{
  "name" : "Richs-MacBook.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "OkZ2-Lj2RjW-pVyVl0C7og",
  "version" : {
    "number" : "7.6.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
    "build_date" : "2020-02-06T00:09:00.449973Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Can someone shed some light on the lack of connectivity to the Elasticsearch host or provide some additional troubleshooting steps I can perform?

Thanks, Rich

Answer 1

As noted by Efrat Levitan the problem was indeed mismatched version. I installed Elasticsearch 7.6.0 and the corresponding Kibana release 7.6.0 and it's working. Fluentd is up and running and logs are showing up in Kibana.