![](/img/trans.png)
[英]Indexing logs with Elasticsearch Logstash (using preprocessing Python script)
[英]save whole database to elasticsearch using logstash
我是ELK的新手,当前正在做的是:
configuration.conf
文件在logstash中设置JDBC(输入>过滤器>输出) input{}
MySQL
查询 pipilines.yml
用于pipilines.yml
配置文件以在单独的线程中运行(即,每个MySQL查询具有(存储在)不同的配置文件中) logstash -f config.conf
(Windows)或仅将logstash
用于管道 我该如何使用logstash获取数据库的所有表并将它们一次索引到ElatsticSeach ,其中每个表的索引与MySQL数据库(Windows)中的表名具有相同的名称。 我可以将查询作为show表运行,获取列表并使用for循环并为每个表定义.conf并将它们另存为.conf文件吗? 但是我该如何修改.yml文件呢? 因为文件是.conf和.yml而不是.py?
官方文档说“每个查询必须具有单独的jdbc”: 配置多个SQL语句
这是脚本:
getTableNames.py
import MySQLdb
# custom made class, Generate
from package_name.generate_conf_yml_logstash import Generate
connection = MySQLdb.connect(host="localhost:3306",
user="root/sa", password="password", db="database_name")
cursor = connection.cursor()
cursor.execute("show tables")
tables = cursor.fetchall()
application_name_tables = []
for table in tables:
application_name_tables.append(table[0])
cursor.close()
connection.close()
Generate.save_files(application_name_tables)
generate_conf_yml_logstash.py
import sys
import os.path
class Generate:
@staticmethod
def save_files(tables):
# save config files
save_path = "D:\folder_name"
for table in tables:
init_f = save_path + "\initial_logstash.conf"
conf_f_name = table + ".conf"
save_file = os.path.join(save_path, conf_f_name)
with open(init_f, "r") as original: # read only
data = original.read()
data = data.replace("table_name", table)
with open(save_file, "w+") as conf: # overwrite if exist
conf.write(data)
# save yml file
yml_f_name = "init_logstash_pipelines.yml"
save_file = os.path.join(save_path, yml_f_name)
with open(save_file, "r") as original:
data = original.read()
with open(save_file, "a+") as yml: # append
for table in tables:
data = data.replace("table", table)
yml.write(data)
sys.exit()
Sample Config和YML文件是:
init_logstash_pipelines.yml
- pipeline.id: table
path.config: "../config/table.conf"
pipeline.workers: 1
initial_logstash.conf
input {
jdbc {
jdbc_driver_library => "../logstash-6.3.0/logstash-core/lib/jars/mysql-connector-java-5.1.46-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://Mysql:3306/database_name"
jdbc_user => "root"
jdbc_password => "password"
statement => "SELECT * from table_name"
# schedule => "* * * * *"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["localhost:9200"]
index => "table_name"
# as every table has diff. primary key, change this please
document_id => "%{pk}"
}
}
请随时更改初始conf,yml文件和代码acc。 根据您的需求
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.