mysql 关系数据库同步到弹性搜索的最佳实践

Question

I have a user field which depends on lot of other fields, it has multiple foreign keys to other table and have many-to-many relation using pivot table and one to many relations.

The API is like the nested JSON structure. The relational data looks like this for a particular user.

user = {
  id : _212dasd,
  name: "alphabeta",
  key1_id = 23,
  key1 : {
     id: 23,
     name: "new name"
  },
  key2 : [
      {
         id: _asdak61526,
         key2_id : 2
         user_id : _212dasd,

      },
      {
         id: _asdak6152a,
         key2_id : 3
         user_id : _212dasd
      }
  ],
  key3: {
     name : "gama",
     key_3_nested : {
       name : "abc",
       key_3_nested_levle2 : {
            name : "xyz",
       }
     }
   }
 }

How can I have mysql relational data synced to elastic search, as most read will go with the elastic search, I am using log-stash for syncing.

input {
  jdbc { 
    jdbc_connection_string => "jdbc:mysql://localhost:3306/database_name"
    # The user we wish to execute our statement as
    jdbc_user => "root"
    jdbc_password => "root"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "/~/mysql-connector-java-5.1.17.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    # our query
    statement => "SELECT * FROM users INNER JOIN table2 ON table2.user_id = users.id" // just a demo query
    schedule => "* * * * * *"
    }
  }
output {
  elasticsearch {
    document_id=> "%{sk_id}.%{user_id}"
    document_type => "doc"
    index => "db_prd"
    hosts => ["http://localhost:9200"]
  }
  stdout{
    codec => json_lines 
  }
}

Question: How can I maintain the relation, is creation multiple join query a good idea as every time it will make a multiple big join on lot of users data and its not optimised structure. For 10000 users and join to multiple tables will slow the sync. Will it sync the every index or only changed one?

How can I get the nested structure on the API from elastic search.

Answer 1

You can use jdbcstream in filter to add Nested object like:

jdbc_streaming {
            jdbc_driver_library => "/~/mysql-connector-java-5.1.17.jar"
            jdbc_driver_class => "org.mariadb.jdbc.Driver"
            jdbc_connection_string =>  "jdbc:mysql://localhost:3306/database_name"
            jdbc_user => "root"
            jdbc_password => "12345678@Abc"
            statement => "SELECT * FROM table2 ON table2.user_id = cus_id"
            parameters => {"cus_id" => "id"}
            target => "table2"
          }

It will create nested "table2" with data is row responsed from statement => "SELECT * FROM table2 ON table2.user_id = cus_id" with "id" in statement => "SELECT * FROM users;"( fix of this statement => "SELECT * FROM users INNER JOIN table2 ON table2.user_id = users.id" )

Notice: It will slow your process