Logstash: Unable to connect to external Amazon RDS Database

Question

Am relatively new to logstash & Elasticsearch...

Installed logstash & Elasticsearch using on macOS Mojave (10.14.2):

brew install logstash
brew install elasticsearch

When I check for these versions:

brew list --versions

Receive the following output:

elasticsearch 6.5.4
logstash 6.5.4

When I open up Google Chrome and type this into the URL Address field:

localhost:9200

This is the JSON response that I receive:

{
    "name" : "9oJAP16",
    "cluster_name" : "elasticsearch_local",
    "cluster_uuid" : "PgaDRw8rSJi-NDo80v_6gQ",
    "version" : {
        "number" : "6.5.4",
        "build_flavor" : "oss",
        "build_type" : "tar",
        "build_hash" : "d2ef93d",
        "build_date" : "2018-12-17T21:17:40.758843Z",
        "build_snapshot" : false,
        "lucene_version" : "7.5.0",
        "minimum_wire_compatibility_version" : "5.6.0",
        "minimum_index_compatibility_version" : "5.0.0"
    },
    "tagline" : "You Know, for Search"
}

Inside:

/usr/local/etc/logstash/logstash.yml

Resides the following variables:

path.data: /usr/local/Cellar/logstash/6.5.4/libexec/data
pipeline.workers: 2
path.config: /usr/local/etc/logstash/conf.d
log.level: info
path.logs: /usr/local/var/log

Inside:

/usr/local/etc/logstash/pipelines.yml

Resides the following variables:

- pipeline.id: main
path.config: "/usr/local/etc/logstash/conf.d/*.conf"

---

Have setup the following logstash_etl.conf file underneath:

/usr/local/etc/logstash/conf.d

Its contents:

input {
    jdbc {
       jdbc_connection_string => "jdbc:mysql://myapp-production.crankbftdpmc.us-west-2.rds.amazonaws.com:3306/products"
       jdbc_user => "products_admin"
       jdbc_password => "products123"
       jdbc_driver_library => "/etc/logstash/mysql-connector/mysql-connector-java-5.1.21.jar"
       jdbc_driver_class => "com.mysql.jdbc.driver"
       schedule => "*/5 * * * *"
       statement => "select * from products"
       use_column_value => false
       clean_run => true
    }
}

# sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-exec
output {
   if ([purge_task] == "yes") {
         exec {
             command => "curl -XPOST 'localhost:9200/_all/products/_delete_by_query?conflicts=proceed' -H 'Content-Type: application/json' -d'
                 {
                   \"query\": {
                     \"range\" : {
                       \"@timestamp\" : {
                         \"lte\" : \"now-3h\"
                       }
                     }
                   }
                 }
             '"
        }
    } 
    else {
        stdout { codec => json_lines}
        elasticsearch {
            "hosts" => "localhost:9200"
            "index" => "product_%{product_api_key}"
            "document_type" => "%{[@metadata][index_type]}"
            "document_id" => "%{[@metadata][index_id]}"
            "doc_as_upsert" => true
            "action" => "update"
            "retry_on_conflict" => 7
        }
    }
}

When I do this:

brew services start logstash

Receive the following inside my /usr/local/var/log/logstash-plain.log file:

[2019-01-15T14:51:15,319][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x399927c7 run>"}
[2019-01-15T14:51:15,663][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-01-15T14:51:16,514][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-01-15T14:57:31,432][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Tried 1 times {:error_message=>"Java::ComMysqlCjJdbcExceptions::CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}
[2019-01-15T14:57:31,435][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Tried 1 times {:error_message=>"Java::ComMysqlCjJdbcExceptions::CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}[2019-01-15T14:51:15,319][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x399927c7 run>"}
[2019-01-15T14:51:15,663][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-01-15T14:51:16,514][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-01-15T14:57:31,432][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Tried 1 times

---

What am I possibly doing wrong?

Is there a way to obtain a dump (eg mysqldump) from an Elasticsearch server (Stage or Production) and then reimport into a local instance running Elasticsearch without using logstash?

This is the same configuration file that works inside an Amazon EC-2 Production Instance but don't know why its not working in my local macOS Mojave instance?

Answer 1

You may encounter the SSL issue of RDS, since

If you use either the MySQL Java Connector v5.1.38 or later, or the MySQL Java Connector v8.0.9 or later to connect to your databases, even if you haven't explicitly configured your applications to use SSL/TLS when connecting to your databases, these client drivers default to using SSL/TLS. In addition, when using SSL/TLS, they perform partial certificate verification and fail to connect if the database server certificate is expired.

as described in AWS RDS Doc

To overcome, either set up the trust store for the LogStash, which is described in the above link as well.

Or take the risk to disable the SSL in the connecting string, like

jdbc_connection_string => "jdbc:mysql://myapp-production.crankbftdpmc.us-west-2.rds.amazonaws.com:3306/products?sslMode=DISABLED"