[英]Logstash can't connect to MySQL database
I'm trying to synchronize the data between the MySQL database and an Elasticsearch using logstash. 我正在尝试使用logstash同步MySQL数据库和Elasticsearch之间的数据。 For that, I created a docker-compose
file that creates containers for Elasticsearch, MySQL and Logstash. 为此,我创建了一个docker-compose
文件,为Elasticsearch,MySQL和Logstash创建容器。
here is the file: 这是文件:
version: '3.1'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
container_name: pelasticsearch
environment:
- discovery.type=single-node
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- elasticsearch:/home/django/elasticsearch
ports:
- 9200:9200
db:
image: mysql
container_name: pmysqldb
command: --default-authentication-plugin=mysql_native_password
restart: always
volumes:
- ${PWD}/pdata/data:/docker-entrypoint-initdb.d
environment:
MYSQL_ROOT_PASSWORD: example
MYSQL_DATABASE: ecommerce
adminer:
image: adminer
restart: always
ports:
- 8080:8080
logstash:
image: docker.elastic.co/logstash/logstash:7.1.1
links:
- elasticsearch
volumes:
- ${PWD}/ls_config:/usr/share/logstash/config/
command: bin/logstash-plugin install logstash-input-jdbc
command: logstash -f /usr/share/logstash/config/logstash.conf
depends_on:
- elasticsearch
volumes:
elasticsearch:
I also had the logstash.yml
and the logstash.conf
files to configure my logstash plugin. 我还有logstash.yml
和logstash.conf
文件来配置我的logstash插件。
input {
jdbc {
jdbc_driver_library => "./mysql-connector-java-5.1.36-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/ecommerce"
jdbc_user => "root"
jdbc_password => "exmaple"
schedule => "* * * * *"
statement => "SELECT * from ecommerce.table;"
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
user => elastic
password => changeme
}
}
---
http.host: '0.0.0.0'
xpack.monitoring.elasticsearch.hosts: [ "elasticsearch:9200"]
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.enabled: true
So, when I run a docker-compose up
all my containers are running including logstash, however when the logstash tries to run the conf file it seems to be unable to load the driver and returns an error as the following: 因此,当我运行docker docker-compose up
时,我的所有容器都在运行,包括logstash,但是当logstash尝试运行conf文件时,似乎无法加载驱动程序并返回错误,如下所示:
logstash_1 | 2056 LogStash::ConfigurationError
logstash_1 | 2056 com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
logstash_1 | 2056 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
Since I'm new to both docker and logstash, is there anything I'm doing wrong? 既然我是docker和logstash的新手,那么我有什么问题吗? and how to solve this error. 以及如何解决这个错误。
So I post this in case someone had the same problem as me. 所以我发布这个以防有人和我有同样的问题。 I found the solution: So the first thing is to use MariaDB
instead of MySQL
, it seems that there are a lot of bugs for MySQL and logstash. 我找到了解决方案:首先是使用MariaDB
而不是MySQL
,似乎MySQL和logstash存在很多错误。 I also created a user instead of connecting as root (this will imply changes to the logstash.conf
file) my docker-compose.yml
target for my MariaDB
is as the following 我还创建了一个用户,而不是以root用户身份连接(这意味着对logstash.conf
文件进行了更改)我的MariaDB
docker-compose.yml
目标如下所示
db:
image: mariadb # mysql
container_name: ph_mariadb
restart: always
volumes:
- ${PWD}/ph_datalake/data/shopware:/docker-entrypoint-initdb.d
environment:
MYSQL_USER: "toto"
MYSQL_PASSWORD: "123456"
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
MYSQL_DATABASE: ecommerce
adminer:
image: adminer
restart: always
ports:
- 8080:8080
Next, copy the jar
file containing the class com.mysql.jdbc.Driver
directly where the jars are located in my container. 接下来,将包含com.mysql.jdbc.Driver
类的jar
文件直接复制到我的容器中的jar所在的位置。 I did that using a volume target on my docker-compose.yml
file in the logstash target so it becomes: 我使用logstash目标中的docker docker-compose.yml
文件上的卷目标执行此操作,因此它变为:
logstash:
image: docker.elastic.co/logstash/logstash:7.1.1
links:
- elasticsearch
- db
volumes:
- ${PWD}/ls_config/logstash.conf:/usr/share/logstash/config/logstash.conf
- ${PWD}/ls_config/logstash.yml:/usr/share/logstash/config/logstash.yml
- ${PWD}/ls_config/mariadb-java-client-2.4.3.jar:/usr/share/logstash/logstash-core/lib/jars/mariadb-java-client-2.4.3.jar
command: bin/logstash-plugin install logstash-input-jdbc
command: logstash -f /usr/share/logstash/config/logstash.conf
depends_on:
- elasticsearch
- db
My logstash.conf
file is also going to change: 我的logstash.conf
文件也将改变:
delete the jdbc_driver_library => "./mysql-connector-java-5.1.36-bin.jar"
from the config file. 从配置文件中删除jdbc_driver_library => "./mysql-connector-java-5.1.36-bin.jar"
。
change the user and password to be the same as the ones on the MariaDB instance. 将用户和密码更改为与MariaDB实例上的用户和密码相同。
So it looks like this: 所以它看起来像这样:
input {
jdbc {
jdbc_driver_class => 'org.mariadb.jdbc.Driver'
jdbc_connection_string => 'jdbc:mariadb://db:3306/ecommerce'
jdbc_user => 'toto'
jdbc_password => '123456'
schedule => "* * * * *"
statement => 'SELECT * from ecommerce.table;'
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
user => elastic
password => changeme
index => "logstash"
}
}
So up to here, my logstash managed to find the class and initiate a Driver instance to connect, but it didn't connect. 到目前为止,我的logstash设法找到了类并启动了一个要连接的Driver实例,但它没有连接。 For some reason the connection was interpreting the address jdbc:mariadb://db:3306/ecommerce
as the localhost of my logstash meaning that when logstash tries to connect with toto
it tries to connect to its own IP address. 由于某种原因,连接是将地址jdbc:mariadb://db:3306/ecommerce
ecommerce解释为我的logstash的本地主机,这意味着当logstash尝试与toto
连接时,它会尝试连接到自己的IP地址。
So the solution to this was to launch only Logstash, and since it depends on db (MariaDB) and elasticsearch it connected $ docker-compose up logstash
所以解决这个问题的方法是只启动Logstash,因为它依赖于db(MariaDB)和$ docker-compose up logstash
它连接了$ docker-compose up logstash
logstash_1 | [2019-08-16T11:22:12,572][INFO ][logstash.inputs.jdbc ] (3.050417s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:23:53,329][INFO ][logstash.inputs.jdbc ] (3.014069s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:25:35,980][INFO ][logstash.inputs.jdbc ] (4.357945s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:27:11,420][INFO ][logstash.inputs.jdbc ] (3.188877s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:28:50,292][INFO ][logstash.inputs.jdbc ] (3.456620s) SELECT * from ecommerce.table;
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.