"Block in Start_input" getting this error while downloading index data from ES to CSV file using Logstash

Question

i'm trying to fetch data from Es index to CSV file using log stash but getting below errors. How to fix this error

[2022-01-07T13:50:27,560][ERROR][logstash.javapipeline ][main][6f7e8b6064fc384f699015f3a0bc42dc623234f39ffeef8f52ed8be2b32fdb29] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::Elasticsearch size=>500, hosts=>["https://lion_jarvis_client:7CaVEWUsjq0wiPmNZT3H@project-prod-api.elastic.services.aws.kpn.org:443/"], query=>"{\r\n "query": {\r\n "bool": {\r\n "must": [],\r\n "filter": [\r\n {\r\n "match_all": {}\r\n }\r\n \r\n ]\r\n }\r\n }\r\n }", scroll=>"5m", index=>"lion-jarvis-netrac-archival-prd-new", docinfo=>true, id=>"6f7e8b6064fc384f699015f3a0bc42dc623234f39ffeef8f52ed8be2b32fdb29", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_db258ee5-69fb-4b41-bc2f-6cd945c0c172", enable_metric=>true, charset=>"UTF-8">, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], connect_timeout_seconds=>10, request_timeout_seconds=>60, socket_timeout_seconds=>60, ssl=>false> Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Exception: Manticore:: ClientProtocolException Stack: C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:37:in block in initialize' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in call' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:274:in call_once' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:158:in code' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in block in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in block in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in search' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:349:in search' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:349:in search_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:272:in do_run_slice' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:250:in do_run' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:238:in run' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:405:in inputworker' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'

My conf file is:

input {
    elasticsearch {
        hosts => "https://lion_jarvis_client:7CaVWDTYjq0wiPmNZT3H@project-api.elastic.services.aws.kpn.org:443/"
        index => "lion-jarvis-netrac-archival-prd-new"
        query => '{
                      "query": {
                          "bool": {
                            "must": [],
                            "filter": [
                              {
                                "match_all": {}
                              }
                            
                            ]
                          }
                        }
                      }'
        size => 500
        scroll => "5m"
        docinfo => true
      
    }
    }
    output {
     

    csv {
      fields => ["ord_no","ord_state","po_no","cust_ord_no","cust_code","service_no"]
      
        path => "\\kpnnl.local\Private\Homedirs2\Homedirs02E\kunch500\logstash1\sample.csv"
      }
    }

Answer 1

The error

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Exception: Manticore::ClientProtocolException Stack

occurs because you are trying to connect to your Elasticsearch Cluster over HTTPS (see the hosts option) but SSL is disabled and you do not provide a Certificate Authority.

Enable SSL with the ssl option of the input plugin:

ssl => true

Next you need to provide a valid filesystem path to the certificate authority (CA) that you used to issue the SSL/TLS certificates for your Elasticsearch nodes:

ca_file => "/your/absolute/path/to/the/ca/file"

So all in all your pipeline configuration should look like this:

input {
    elasticsearch {
        hosts => "https://lion_jarvis_client:7CaVWDTYjq0wiPmNZT3H@project-api.elastic.services.aws.kpn.org:443/"
        index => "lion-jarvis-netrac-archival-prd-new"
        query => '{
                  "query": {
                      "bool": {
                        "must": [],
                        "filter": [
                          {
                            "match_all": {}
                          }
                        
                        ]
                      }
                    }
                  }'
        size => 500
        scroll => "5m"
        docinfo => true
        ssl => true
        ca_file => "/your/absolute/path/to/the/ca/file"  
    }
}
output {
    csv {
        fields => ["ord_no","ord_state","po_no","cust_ord_no","cust_code","service_no"]
        path => "\\kpnnl.local\Private\Homedirs2\Homedirs02E\kunch500\logstash1\sample.csv"
    }
}

I hope I could help you.