无法使用 Logstash 将数据导入 AWS Elasticsearch 实例

Question

I'm trying to import a CSV file to my AWS Elasticsearch ( v7.10 ) with Logstash ( v7.13 ) from my local system. Locally all the imports are working, but while uploading on AWS it is giving me error:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to D:/logstash-7.13.3/logs which is now configured via log4j2.properties
[2021-07-21T17:45:20,891][INFO ][logstash.runner          ] Log4j configuration path used is: D:\logstash-7.13.3\config\log4j2.properties
[2021-07-21T17:45:20,906][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.3", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [mswin32-x86_64]"}
[2021-07-21T17:45:21,078][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-07-21T17:45:22,356][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-07-21T17:45:32,539][INFO ][org.reflections.Reflections] Reflections took 94 ms to scan 1 urls, producing 24 keys and 48 values
[2021-07-21T17:45:36,529][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443"]}
[2021-07-21T17:45:37,482][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://noetic:xxxxxx@XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443/]}}
[2021-07-21T17:45:38,211][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://noetic:xxxxxx@XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443/"}
[2021-07-21T17:45:38,320][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.10.2) {:es_version=>7}
[2021-07-21T17:45:38,320][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-07-21T17:45:38,383][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"https://noetic:xxxxxx@XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443/_license'"}
[2021-07-21T17:45:38,414][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://noetic:xxxxxx@XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443/"}
[2021-07-21T17:45:38,612][ERROR][logstash.outputs.elasticsearch][main] Unable to retrieve Elasticsearch cluster uuid {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :backtrace=>["D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:381:in `with_connection'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:292:in `perform_request'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:300:in `block in get'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:199:in `get'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:152:in `discover_cluster_uuid'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch.rb:308:in `finish_register'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch.rb:279:in `block in register'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:145:in `block in after_successful_connection'"]}
[2021-07-21T17:45:38,750][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-07-21T17:45:38,984][ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :backtrace=>["D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:381:in `with_connection'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:292:in `perform_request'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:300:in `block in Pool'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:379:in `exists?'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:384:in `template_exists?'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:80:in `template_install'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch.rb:496:in `install_template'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch.rb:309:in `finish_register'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch.rb:279:in `block in register'", "D:/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:145:in `block in after_successful_connection'"]}

My output config is:

output {

    # stdout { codec =>  "rubydebug" }
    
  elasticsearch {
    hosts => ["https://XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443"]
    ssl => true
    index => "em-index-csv"
    user => "noetic"
    password => "XXXXXXXXX"
    ilm_enabled => false
  }
}

Is this a version issue? Do I need to install Logstash with v7.10 similar to the AWS Elasticsearch version?

Answer 1

Logstash should never be newer than Elasticsearch.

It's often okay for Logstash to lag behind Elasticsearch.. but not the other way around. See When not to upgrade for a docs reference.

This log message points at another potential issue:

Got response code '401' contacting Elasticsearch at URL 'https://XXXXX.xxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443/_license'

The OSS version of Logstash is more likely to play nice with AWS. Any features available in the non-OSS (default / Elastic License) version may not be supported by AWS, which runs OSS versions of Elasticsearch.