Logstash adding multiple configs for delta and fulltruth indexing from oracle database

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. Geo Thomas is looking for an . 。

I was able to create an elasticsearch index from an oracle database using the following logstash.conf

input {

 jdbc
{

    jdbc_driver_library => "<path>/ojdbc10.jar"

    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"

    jdbc_connection_string => "connection string"

    jdbc_user => <username>

    jdbc_password => <pwd>

    statement => "SELECT * FROM my table name WHERE last_modified_date >= :sql_last_value"

    schedule => "* * * * *"
"

}
}
output {
  elasticsearch {
     hosts => ["http://localhost:9200"]
        index => "test2"
        document_id => "%{col1}-%{col2}-%{col3}"
        action => "update"
      doc_as_upsert => true
      document_id => "%{col1}-%{col2}-%{col3}"


  }

But this will always run a fulltruth. I am now trying to run this as a delta may be every hour and the fulltruth overnight so that I will have clean data everyday. Also I dont want these two executions to confilct with each other. So I created two config files as below:

delta.conf

input {

 jdbc
{

    jdbc_driver_library => "<path>/ojdbc10.jar"

    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"

    jdbc_connection_string => "connection string"

    jdbc_user => <username>

    jdbc_password => <pwd>

    statement => "SELECT * FROM my table name WHERE last_modified_date >= :sql_last_value"

    schedule => "*/15 * * * *"
"

}

filter {

  if !("$FULL_IMPORT_RUNNING" in [tags])
    {

   
    }
    else {

    drop {}

    }

  }


}
output 
{

     if !("$FULL_IMPORT_RUNNING" in [tags]) 
     {

        elasticsearch 
        {
            hosts => ["http://localhost:9200"]
                index => "test2"
                document_id => "%{col1}-%{col2}-%{col3}"
                action => "update"
            doc_as_upsert => true
            document_id => "%{col1}-%{col2}-%{col3}"
        }
     }

}

full.conf:

input {

 jdbc
{

    jdbc_driver_library => "<path>/ojdbc10.jar"

    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"

    jdbc_connection_string => "connection string"

    jdbc_user => <username>

    jdbc_password => <pwd>

    statement => "SELECT * FROM my table name"

    schedule => "0 0 * * *"
"

}

filter {

  mutate {

    add_tag => ["$FULL_IMPORT_RUNNING"]

  }

}



}
output 
{


elasticsearch 
{
    hosts => ["http://localhost:9200"]
        index => "test2"
        document_id => "%{col1}-%{col2}-%{col3}"
        action => "update"
    doc_as_upsert => true
    document_id => "%{col1}-%{col2}-%{col3}"
}
    

}

And I am using the following command to start my logstash:

./logstash -f../config/delta.conf -f../config/full.conf

I see that logstash is starting up, but nothing happening after that. If I start logstash with only one config file, I see its working. Am I doing something wrong here?

Answer 1

You cannot run Logstash with two config files using the -f command line flag. You have two options.

Option A: Use multiple pipelines and specify both of them in the pipelines.yml file.

- pipeline.id: full
  path.config: "/etc/path/to/full.conf"
- pipeline.id: delta
  path.config: "/etc/path/to/delta.conf"

Option B: Combine both configs into a single config file and add conditional processing of the events based on tags. Then run the config with -f

input {
  # Full processing
  jdbc {
    ...
    tags => ["FULL_IMPORT_RUNNING"]
  }
  # Delta processing
  jdbc {
    ...
    tags => ["DELTA_IMPORT_RUNNING"]
  }
}
filter {
  if "FULL_IMPORT_RUNNING" in [tags] {
    ...
  }
  if "DELTA_IMPORT_RUNNING" in [tags] {
    ...
  }
}
output {
  if "FULL_IMPORT_RUNNING" in [tags] {
    ...
  }
  if "DELTA_IMPORT_RUNNING" in [tags] {
    ...
  }
}