Filebeat 6.7.0 not pushing logs to Elasticsearch 6.7.0 after upgrade

Question

Filebeat stopped working after upgrading from 6.6.2 to 6.7.0

My Filebeat configuration is:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/www/current/log/production.log
    - /var/www/current/log/api_v2.production.log
  multiline.pattern: '^[EIWDF]\, '
  multiline.negate: true
  multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.elasticsearch:
  hosts: ["ip:29200"]

But /var/log/filebeat/filebeat shows:

2019-03-28T14:12:11.091Z    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://ip:9200)): Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats
2019-03-28T14:12:11.091Z    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-03-28T14:12:11.091Z    INFO    [publish]   pipeline/retry.go:191     done
2019-03-28T14:12:11.091Z    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2019-03-28T14:12:11.091Z    INFO    [publish]   pipeline/retry.go:168     done
2019-03-28T14:12:11.091Z    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://ip:9200)) with 5 reconnect attempt(s)
2019-03-28T14:12:11.125Z    INFO    elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 6.7.0

However when i hit curl ip:9200 it return:

{
  "name" : "30KRsiU",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "xim_BCzFSXWrAO_kMO3TQA",
  "version" : {
    "number" : "6.7.0",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "8453f77",
    "build_date" : "2019-03-21T15:32:29.844721Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

How do I get this working? I am out of ideas. Thanks for any help.

Answer 1

If you are using the open source "OSS" distribution of Elasticsearch, then all connected beats must also be deployed with the corresponding "OSS" version. When using docker images, append the -oss suffix to the image name to get the correct image. For the file based downloads from the elastic website, insert -oss before the version number, eg https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.0.0-linux-x86_64.tar.gz

Answer 2

I've got the same issue ( with heartbeat) and one suggested solution is downgrade to 6.6.2

2019-04-01T09:36:27.474Z ERROR instance/beat.go:802 Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://localhost:19200 : Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats]

Steps

wget https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-6.6.2-amd64.deb

dpkg -i heartbeat-6.6.2-amd64.deb

sudo service heartbeat-elastic restart

Log now showed it successfully connected to ES

2019-04-01T09:42:38.061Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch( http://localhost:9200 )) established