Logstash Mysql configuration

Question

I am using Logstash to pull data from Mysql into Elastic. The configuration is something like

 input {
  jdbc {
    jdbc_driver_library => "/usr/share/mysql-connector-java-5.1.46-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://mysql:3306/books"
    jdbc_user => "root"
    jdbc_password=>"1"
    schedule => "* * * * *"
    statement => "SELECT * FROM book"
  }
}

So the pulling query is just,

  statement => "SELECT * FROM book"

which is shown in most of the documentations. The question is, will logstash understand what has changed and what needs to be indexed/reindexed, or is it just going to scan the full table and reindex everything? For my case, the data in the table is going to be pretty much static, however, it should be immediately (at least as soon as possible) available once a record is created. So, the pulling interval is low. Should I create a more complicated query for pulling or should I assume Logstash will do the trick?

Answer 1

Using your configuration logstash will fetch all the records selected by the query SELECT * FROM book.

If you want to read only new record, you must set in the configuration an incremental column that will be used as a where in the configuration:

jdbc {
    statement => "SELECT id, mycolumn1, mycolumn2 FROM my_table WHERE id > :sql_last_value"
    use_column_value => true
    tracking_column => "id"
    # ... other configuration bits
  }

The setting tracking_column will enable the feature of storing the last value of the column specified in tracking_column. The last value :sql_last_value can be used in the query as a custom condition in the where. The logstash state will be stored on the file-system, and only few metadata related to the last value are memorized.

Notice that in that way you can fetch only the new inserted records, but depending on your data-model it cannot fetch the deletion or the update of already imported data.