Logstash 管道停止将数据插入 elasticsearch

Question

尽管管道正在运行，但 Logstash 管道并未将数据摄取到 elasticsearch 索引中。 这条管道是一年前部署的，从那时起它运行良好。 但在 2021 年 5 月 24 日，它停止了提取数据。 重新启动 logstash 解决了这个问题。 我们检查了 logstash 日志，但没有发现任何东西。 请查看以下日志。

Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,503][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>http://localhost:9200/}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,619][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,623][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Aug 26 12:24:22 xyz.com systemd[1]: Stopping logstash...
Aug 26 12:24:22 xyz.com logstash[827]: [2020-08-26T12:24:22,665][WARN ][logstash.runner          ] SIGTERM received. Shutting down.
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,864][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>33, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.9-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>25, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>26, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>27, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>28, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>29, "name"=>"[main]>worker4", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>31, "name"=>"[main]>worker6", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>32, "name"=>"[main]>worker7", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,866][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,852][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x2c879c55 run>"}
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,855][INFO ][logstash.runner          ] Logstash shut down.
Aug 26 12:24:29 xyz.com systemd[1]: Stopped logstash.
May 24 08:11:46 xyz.com systemd[1]: Started logstash.
May 24 08:12:11 xyz.com logstash[19174]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 08:12:11 xyz.com logstash[19174]: 2021-05-24 08:12:11,794 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
May 24 08:12:11 xyz.com logstash[19174]:         at java.io.FileOutputStream.open0(Native Method)
May 24 08:12:11 xyz.com logstash[19174]:         at java.io.FileOutputStream.open(FileOutputStream.java:270)
May 24 08:12:11 xyz.com logstash[19174]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
May 24 08:12:11 xyz.com logstash[19174]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
May 24 08:12:11 xyz.com logstash[19174]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
 
 
 
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,120][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,363][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Main process exited, code=killed, status=9/KILL
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Failed with result 'signal'.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
May 24 15:38:50 xyz.com systemd[1]: Stopped logstash.
May 24 15:38:50 xyz.com systemd[1]: Started logstash.
May 24 15:39:06 xyz.com logstash[25666]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 15:39:07 xyz.com logstash[25666]: 2021-05-24 15:39:07,101 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFo
May 24 15:39:07 xyz.com logstash[25666]:         at java.io.FileOutputStream.open0(Native Method)
May 24 15:39:07 xyz.com logstash[25666]:         at java.io.FileOutputStream.open(FileOutputStream.java:270)

服务器操作系统：Ubuntu 18.04 ELK 版本：7.11.1

需要您的帮助以找出确切的原因。

Answer 1

您是否尝试将捆绑包更新到较新的版本