Logstash (6.5.4) Adding pipeline for elasticsearch

Question

Logs format jobID status data .

Let's say my logs are printed in order 5hgsxyt3838 RUNNING data 5hgsxyt3838 RUNNING data 5hgsxyt3838 COMPLETE data . Here I want my final status at elasticsearch to be the status in the last line of the log. But I have observed that for multiple jobID's my final status is not the status in the last line but from some other previous line. What may be the reason? Is there a way to fix this using pipeline (if my final line status is COMPLETE, any other previous line should not override it) or something else?

Output block

   output {
    if [fields][target_index] == "import-export-logger" {
        if [status] == "SCHEDULED" {
            elasticsearch {
                hosts => [ "localhost:9200" ]
                index => "import-export-logger-%{index-name}"
                document_id => "%{jobID}"
                action => "create"
            }
        }
        else {
            elasticsearch {
                hosts => [ "localhost:9200" ]
                index => "import-export-logger-%{index-name}"
                document_id => "%{jobID}"
                action => "update"
                doc_as_upsert => true
            }
        }
    }
}

Answer 1

You need to run your pipeline with only one worker. If you don't, as many worker threads are created as the number of available CPUs you have at your disposal, and in this case, some events might be processed concurrently and a previous line might arrive after the last one.

If you have configured your pipeline in pipeline.yml , you need to set this in your configuration file:

pipeline.workers: 1

Otherwise, if you run Logstash from the command line, you simply need to add the -w switch

bin/logstash -f logstash.conf -w 1

or equivalently

bin/logstash -f logstash.conf --pipeline.workers 1