Logstash s3 輸出寫入格式 logstash s3 輸入不理解

Question

我正在從兩個獨立的 ES 集群中移動數據。 我已將 s3 添加為公共區域，並有兩個 logstash 實例，一個從 Elasticsearch 寫入 s3，另一個讀取 S3 並加載 Elasticsearch。

問題是每個索引中只加載一個文檔。 s3 輸出插件編寫的輸出文件是一行很長的文件，許多 json 文檔都運行在一起，沒有逗號，也沒有為數組打開或關閉方括號。 例如，而不是[{"id":1},{"id":2},{"id":3}]輸出正在寫入讀取{"id":1}{"id":2}{"id":3} 。 在這種情況下，logstash 僅使用 s3 作為輸入讀取{"id":1} 。

去s3的配置是：

input {
  elasticsearch {
    hosts => ["${ES_HOST}:${ES_PORT}"]
    index => "${ES_INDEX}"
    password => "${ES_PASS}"
    ssl => "true"
    user => "${ES_USER}"
}
  }
output {
  s3 {
      bucket => "${S3_BUCKET}"
      encoding => "gzip"
      codec => "json"
      prefix => "${S3_PREFIX}/${ES_INDEX}"
      region => "ap-southeast-2"
    }
}

S3的配置讀數為：

input {
  s3 {
      bucket => "${S3_BUCKET}"
      codec => "json"
      prefix => "${S3_PREFIX}/${ES_INDEX}/"
      region => "ap-southeast-2"
      watch_for_new_files => false
    }
  }
output {
  stdout { }
}

在這兩種情況下， ${}變量都是在環境（bash shell）中設置的。

兩台服務器都運行logstash 7.6.0

PS：我認為它們並不重要，但是來自 logstash 的 stdout 日志說：

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/ec2-user/logstash-7.6.0/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /home/ec2-user/logstash-7.6.0/logs which is now configured via log4j2.properties
[2020-03-09T01:10:35,168][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-03-09T01:10:35,353][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.0"}
[2020-03-09T01:10:37,813][INFO ][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 20 keys and 40 values 
[2020-03-09T01:10:53,476][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-03-09T01:10:53,515][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/home/ec2-user/kibana/from_s3.conf"], :thread=>"#<Thread:0x1364485f run>"}
[2020-03-09T01:10:54,561][INFO ][logstash.inputs.s3       ][main] Registering s3 input {:bucket=>"my-bucket-here", :region=>"ap-southeast-2"}
[2020-03-09T01:10:55,334][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-03-09T01:10:55,435][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-03-09T01:10:55,833][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-03-09T01:10:57,507][INFO ][logstash.inputs.s3       ][main] Using default generated file for the sincedb {:filename=>"/home/ec2-user/logstash-7.6.0/data/plugins/inputs/s3/sincedb_1906e463a09b003733b719c08277c793"}
/home/ec2-user/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
my-document-here
}
[2020-03-09T01:10:59,587][INFO ][logstash.runner          ] Logstash shut down.

PPS：刪除由於DB允許加載一行，文件沒有改變。

Answer 1

在您的 s3 輸出插件中使用以下內容。

編解碼器 => "json_lines"

這確實用新行分隔每個事件。

s3 輸入插件仍然可以使用“json”的編解碼器。