Logstash with jdbc driver-How to set an initial value for sql_last_value

Question

According to Logstash documentation the sql_last_value parameter is set to Thursday, 1 January 1970 before any query is run, and stored in the YAML file according to the last_run_metadata_path parameter. I need to set this value to a custom value, but because I used Logstash inside a Docker container, I wasn't able to use a custom image with a file initialized with the desired value because the file is read-only in the container and then Logstash throws an error. My question is: is it possible to start Logstash with a different initial value than the default in a container installation? Thanks.

Answer 1

You can have a workaround. Logstash allows you to define a file in which it will store the last value that sql_last_value had. When logstash starts the jdbc will look in that file (if defined in the settings), store the value to the sql_last_value parameter and continue from there.

You can create that file and write the date you want to start with. That way the driver will think it stopped there the last time. Here's how you can do it:

In the logstash.conf add the following:

input {
  jdbc {
    last_run_metadata_path => "/some/path/sql_last_value.yml"
    # ... other configuration bits
  }
}

Write the date you want to start with to the sql_last_value.yml file in the following format:

--- 2020-06-12 20:28:00.374000000 Z

You can easily create the file with a simple command in your dockerfile.