[英]Logstash can't connect to MySQL database
我正在尝试使用logstash同步MySQL数据库和Elasticsearch之间的数据。 为此,我创建了一个docker-compose
文件,为Elasticsearch,MySQL和Logstash创建容器。
这是文件:
version: '3.1'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
container_name: pelasticsearch
environment:
- discovery.type=single-node
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- elasticsearch:/home/django/elasticsearch
ports:
- 9200:9200
db:
image: mysql
container_name: pmysqldb
command: --default-authentication-plugin=mysql_native_password
restart: always
volumes:
- ${PWD}/pdata/data:/docker-entrypoint-initdb.d
environment:
MYSQL_ROOT_PASSWORD: example
MYSQL_DATABASE: ecommerce
adminer:
image: adminer
restart: always
ports:
- 8080:8080
logstash:
image: docker.elastic.co/logstash/logstash:7.1.1
links:
- elasticsearch
volumes:
- ${PWD}/ls_config:/usr/share/logstash/config/
command: bin/logstash-plugin install logstash-input-jdbc
command: logstash -f /usr/share/logstash/config/logstash.conf
depends_on:
- elasticsearch
volumes:
elasticsearch:
我还有logstash.yml
和logstash.conf
文件来配置我的logstash插件。
input {
jdbc {
jdbc_driver_library => "./mysql-connector-java-5.1.36-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/ecommerce"
jdbc_user => "root"
jdbc_password => "exmaple"
schedule => "* * * * *"
statement => "SELECT * from ecommerce.table;"
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
user => elastic
password => changeme
}
}
---
http.host: '0.0.0.0'
xpack.monitoring.elasticsearch.hosts: [ "elasticsearch:9200"]
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.enabled: true
因此,当我运行docker docker-compose up
时,我的所有容器都在运行,包括logstash,但是当logstash尝试运行conf文件时,似乎无法加载驱动程序并返回错误,如下所示:
logstash_1 | 2056 LogStash::ConfigurationError
logstash_1 | 2056 com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
logstash_1 | 2056 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
既然我是docker和logstash的新手,那么我有什么问题吗? 以及如何解决这个错误。
所以我发布这个以防有人和我有同样的问题。 我找到了解决方案:首先是使用MariaDB
而不是MySQL
,似乎MySQL和logstash存在很多错误。 我还创建了一个用户,而不是以root用户身份连接(这意味着对logstash.conf
文件进行了更改)我的MariaDB
docker-compose.yml
目标如下所示
db:
image: mariadb # mysql
container_name: ph_mariadb
restart: always
volumes:
- ${PWD}/ph_datalake/data/shopware:/docker-entrypoint-initdb.d
environment:
MYSQL_USER: "toto"
MYSQL_PASSWORD: "123456"
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
MYSQL_DATABASE: ecommerce
adminer:
image: adminer
restart: always
ports:
- 8080:8080
接下来,将包含com.mysql.jdbc.Driver
类的jar
文件直接复制到我的容器中的jar所在的位置。 我使用logstash目标中的docker docker-compose.yml
文件上的卷目标执行此操作,因此它变为:
logstash:
image: docker.elastic.co/logstash/logstash:7.1.1
links:
- elasticsearch
- db
volumes:
- ${PWD}/ls_config/logstash.conf:/usr/share/logstash/config/logstash.conf
- ${PWD}/ls_config/logstash.yml:/usr/share/logstash/config/logstash.yml
- ${PWD}/ls_config/mariadb-java-client-2.4.3.jar:/usr/share/logstash/logstash-core/lib/jars/mariadb-java-client-2.4.3.jar
command: bin/logstash-plugin install logstash-input-jdbc
command: logstash -f /usr/share/logstash/config/logstash.conf
depends_on:
- elasticsearch
- db
我的logstash.conf
文件也将改变:
从配置文件中删除jdbc_driver_library => "./mysql-connector-java-5.1.36-bin.jar"
。
将用户和密码更改为与MariaDB实例上的用户和密码相同。
所以它看起来像这样:
input {
jdbc {
jdbc_driver_class => 'org.mariadb.jdbc.Driver'
jdbc_connection_string => 'jdbc:mariadb://db:3306/ecommerce'
jdbc_user => 'toto'
jdbc_password => '123456'
schedule => "* * * * *"
statement => 'SELECT * from ecommerce.table;'
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
user => elastic
password => changeme
index => "logstash"
}
}
到目前为止,我的logstash设法找到了类并启动了一个要连接的Driver实例,但它没有连接。 由于某种原因,连接是将地址jdbc:mariadb://db:3306/ecommerce
ecommerce解释为我的logstash的本地主机,这意味着当logstash尝试与toto
连接时,它会尝试连接到自己的IP地址。
所以解决这个问题的方法是只启动Logstash,因为它依赖于db(MariaDB)和$ docker-compose up logstash
它连接了$ docker-compose up logstash
logstash_1 | [2019-08-16T11:22:12,572][INFO ][logstash.inputs.jdbc ] (3.050417s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:23:53,329][INFO ][logstash.inputs.jdbc ] (3.014069s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:25:35,980][INFO ][logstash.inputs.jdbc ] (4.357945s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:27:11,420][INFO ][logstash.inputs.jdbc ] (3.188877s) SELECT * from ecommerce.table;
logstash_1 | [2019-08-16T11:28:50,292][INFO ][logstash.inputs.jdbc ] (3.456620s) SELECT * from ecommerce.table;
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.