Logstash can't connect to MySQL database

Question

I'm trying to synchronize the data between the MySQL database and an Elasticsearch using logstash. For that, I created a docker-compose file that creates containers for Elasticsearch, MySQL and Logstash.

here is the file:

docker-compose.yml

version: '3.1'

services:
   elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
     container_name: pelasticsearch
     environment:
     - discovery.type=single-node
     - cluster.name=docker-cluster
     - bootstrap.memory_lock=true
     - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     ulimits:
       memlock:
         soft: -1
         hard: -1
     volumes:
     - elasticsearch:/home/django/elasticsearch
     ports:
     - 9200:9200

   db:
    image: mysql
    container_name: pmysqldb
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    volumes:
      - ${PWD}/pdata/data:/docker-entrypoint-initdb.d
    environment:
      MYSQL_ROOT_PASSWORD: example
      MYSQL_DATABASE: ecommerce

   adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

   logstash:
       image: docker.elastic.co/logstash/logstash:7.1.1
       links:
       - elasticsearch
       volumes:
       - ${PWD}/ls_config:/usr/share/logstash/config/
       command: bin/logstash-plugin install logstash-input-jdbc
       command: logstash -f /usr/share/logstash/config/logstash.conf
       depends_on:
       - elasticsearch

volumes:
  elasticsearch:

I also had the logstash.yml and the logstash.conf files to configure my logstash plugin.

logstash.conf

input {
  jdbc {
    jdbc_driver_library => "./mysql-connector-java-5.1.36-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/ecommerce"
    jdbc_user => "root"
    jdbc_password => "exmaple"
    schedule => "* * * * *"
    statement => "SELECT * from ecommerce.table;"
  }
}
output {
   elasticsearch {
        hosts => ["elasticsearch:9200"]
        user => elastic
        password => changeme
   }
}

logstash.yml

---
http.host: '0.0.0.0'
xpack.monitoring.elasticsearch.hosts: [ "elasticsearch:9200"]
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.enabled: true

So, when I run a docker-compose up all my containers are running including logstash, however when the logstash tries to run the conf file it seems to be unable to load the driver and returns an error as the following:

logstash_1       |   2056     LogStash::ConfigurationError
logstash_1       |   2056     com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
logstash_1       |   2056       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'

Since I'm new to both docker and logstash, is there anything I'm doing wrong? and how to solve this error.

Answer 1

So I post this in case someone had the same problem as me. I found the solution: So the first thing is to use MariaDB instead of MySQL , it seems that there are a lot of bugs for MySQL and logstash. I also created a user instead of connecting as root (this will imply changes to the logstash.conf file) my docker-compose.yml target for my MariaDB is as the following

   db:
    image: mariadb # mysql
    container_name: ph_mariadb
    restart: always
    volumes:
      - ${PWD}/ph_datalake/data/shopware:/docker-entrypoint-initdb.d
    environment:
      MYSQL_USER: "toto"
      MYSQL_PASSWORD: "123456"
      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
      MYSQL_DATABASE: ecommerce

   adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

Next, copy the jar file containing the class com.mysql.jdbc.Driver directly where the jars are located in my container. I did that using a volume target on my docker-compose.yml file in the logstash target so it becomes:

   logstash:
       image: docker.elastic.co/logstash/logstash:7.1.1
       links:
       - elasticsearch
       - db
       volumes:
       - ${PWD}/ls_config/logstash.conf:/usr/share/logstash/config/logstash.conf
       - ${PWD}/ls_config/logstash.yml:/usr/share/logstash/config/logstash.yml
       - ${PWD}/ls_config/mariadb-java-client-2.4.3.jar:/usr/share/logstash/logstash-core/lib/jars/mariadb-java-client-2.4.3.jar 
       command: bin/logstash-plugin install logstash-input-jdbc
       command: logstash -f /usr/share/logstash/config/logstash.conf
       depends_on:
       - elasticsearch
       - db

My logstash.conf file is also going to change:

• delete the jdbc_driver_library => "./mysql-connector-java-5.1.36-bin.jar" from the config file.

• change the user and password to be the same as the ones on the MariaDB instance.

So it looks like this:

input {
  jdbc {
    jdbc_driver_class => 'org.mariadb.jdbc.Driver'
    jdbc_connection_string => 'jdbc:mariadb://db:3306/ecommerce'
    jdbc_user => 'toto'
    jdbc_password => '123456'
    schedule => "* * * * *"
    statement => 'SELECT * from ecommerce.table;'
  }
}
output {
   elasticsearch {
        hosts => ["elasticsearch:9200"]
        user => elastic
        password => changeme
        index => "logstash"
   }
}

So up to here, my logstash managed to find the class and initiate a Driver instance to connect, but it didn't connect. For some reason the connection was interpreting the address jdbc:mariadb://db:3306/ecommerce as the localhost of my logstash meaning that when logstash tries to connect with toto it tries to connect to its own IP address.

So the solution to this was to launch only Logstash, and since it depends on db (MariaDB) and elasticsearch it connected $ docker-compose up logstash

logstash_1       | [2019-08-16T11:22:12,572][INFO ][logstash.inputs.jdbc     ] (3.050417s) SELECT * from ecommerce.table;
logstash_1       | [2019-08-16T11:23:53,329][INFO ][logstash.inputs.jdbc     ] (3.014069s) SELECT * from ecommerce.table;
logstash_1       | [2019-08-16T11:25:35,980][INFO ][logstash.inputs.jdbc     ] (4.357945s) SELECT * from ecommerce.table;
logstash_1       | [2019-08-16T11:27:11,420][INFO ][logstash.inputs.jdbc     ] (3.188877s) SELECT * from ecommerce.table;
logstash_1       | [2019-08-16T11:28:50,292][INFO ][logstash.inputs.jdbc     ] (3.456620s) SELECT * from ecommerce.table;