使用logstash同步数据

Question

I'm trying to use logstash to sync all my data on my MySql server to my Elasticsearch server.

I've aleardy learned the basics of logstash.conf, this is my file:

input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://localhost/homestead"
        jdbc_user => "homestead"
        jdbc_password => "secret"
        jdbc_driver_library => "/home/vagrant/Code/mysql-connector-java-5.1.38-bin.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        statement => "SELECT * from volunteer"
    }
    jdbc {
        jdbc_connection_string => "jdbc:mysql://localhost/homestead"
        jdbc_user => "homestead"
        jdbc_password => "secret"
        jdbc_driver_library => "/home/vagrant/Code/mysql-connector-java-5.1.38-bin.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        statement => "SELECT * from contact"
    }

} 
output {
    elasticsearch {
        document_id => "%{uid}"
        hosts => "localhost"
    } 
}

My intention is to copy every table into a Type. How do I specify this?

edit: "type" instead of "index"

Thank you!

Answer 1

What you can do is simply to add a field (using add_field ) in each input denoting the type name which you want the data to be indexed in and then use that variable as the type name in the elasticsearch output.

input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://localhost/homestead"
        jdbc_user => "homestead"
        jdbc_password => "secret"
        jdbc_driver_library => "/home/vagrant/Code/mysql-connector-java-5.1.38-bin.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        statement => "SELECT * from volunteer"
        add_field => {"type" => "volunteer"}
    }
    jdbc {
        jdbc_connection_string => "jdbc:mysql://localhost/homestead"
        jdbc_user => "homestead"
        jdbc_password => "secret"
        jdbc_driver_library => "/home/vagrant/Code/mysql-connector-java-5.1.38-bin.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        statement => "SELECT * from contact"
        add_field => {"type" => "contact"}
    }
} 
output {
    elasticsearch {
        hosts => ["localhost"]
        index => "homestead"
        document_type => "%{type}"             <--- specify the index here
        document_id => "%{uid}"
    } 
}

Be aware though that using the same index to host several different mapping types might lead to type conflicts. The short story is that two different fields with the same name in two different types MUST ALWAYS have the same type definition. Read more about it in this blog article