Logstash with Elasticsearch: I am using logstash to ingest the data into the ES index. But now I want logstash to run 24/7

Question

    #file:db.conf

input { 
    jdbc { 
           jdbc_driver_library => ""
           jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver" 
           jdbc_connection_string => "jdbc:oracle:thin:@abcd.klm.uvw:1521/qtp1"
           jdbc_user =>"user_wew"
           jdbc_password => "password_wew"

           
           
                  statement => "select col1, col2, col3, col4, col5, col6, countid,max(version) as mv from master_object_table where version >:sql_last_value group by countid"

            schedule => "* * * * *"
           last_run_metadata_path => "C:/ES1/ELK_stack_7.4.2/logstash-7.4.2/logstash-7.4.2/Master_refresh_a.txt"

           use_column_value => true

            tracking_column => "version"

         } 
       }
       
        filter {
      mutate {
        convert => {
          "countid" => "string"
          }
          }
          }
       
output {
  elasticsearch { 
           hosts => "localhost:9200" 
           index =>"refresh_index_a"
           document_id =>"%{countid}"
           #document_type="_doc"
           
           }
 file {
         path => "C:\\ES1\\ELK_stack_7.4.2\\logstash-7.4.2\\logstash-7.4.2\\bin\\logstashESRecordsIngestionDetails_refresh_a.txt"
         
         codec => rubydebug
       }
           
  stdout {   codec => rubydebug }
}

Above is my logstash config file setting. I want to run this logstash 24/7 and also if the machine shutsdown on which this logstash is running then how can I manage that as this logstash is ingesting the live data to ES index. Please suggest. Is there any way if one server goes down the logstash on another node will continue the work.

Answer 1

As per the documentation

Logstash is horizontally scalable and can form groups of nodes running the same pipeline. Logstash's adaptive buffering capabilities will facilitate smooth streaming even through variable throughput loads. If the Logstash layer becomes an ingestion bottleneck, simply add more nodes to scale out. Here are a few general recommendations:

1. Beats should load balance across a group of Logstash nodes.
2. A minimum of two Logstash nodes are recommended for high availability.
3. It's common to deploy just one Beats input per Logstash node, but multiple Beats inputs can also be deployed per Logstash node to expose independent endpoints for different data sources.