FileBeat harvesting issues

Question

We are using ELK for controlling our program logs. In our FileBeat config we are harvesting from 30 different paths which contains files that updates every second (it updates every second only in the prod's machines - in the other Dev machines we have significantly less logs). Our log files not get deleted until they getting old and we stop using them (also we don't modify there names) . Lately we found out that the logs from last paths in the configuration file (.yml) from the prod machines is never appearing in the Kibana.

After investigation, we realized that FileBeat stuck on the files is the first's path's and never seem to reach the last one's. When I replace the location of the last two paths to the beginning, FileBeat started to register all the logs there and later on harvest them.

I looked up in the documentation on the FileBeat configuration and I saw the close* options close_option_config which seem like a good idea. But I didn't managed to get it right yet and I don't sure what is the recommended time for the scan_frequency option (that for now is default of 10s) and what would serve me in the best way.

I tried to change the close_timeout to 15s and the scan_frequency to 2m

      close_timeout: 15s
      scan_frequency: 2m

I would like to here some opinion what can I do to get solve this problem? I put the config here to have some reference and to see if I missed something else.

my filebeat.yml: (before changes)

      filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - D:\logs\*\path1\a_*_Pri_app.log.txt
      input_type: log
      document_type: type1
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path2\b_*_Paths_app.log.txt
      input_type: log
      document_type: type2
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path3\c_*_R_app.log.txt
      input_type: log
      document_type: path3
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path4\d_*_d_app.log.txt
        - C:\logs\*\path4\d_*_d_app.log.txt
      input_type: log
      document_type: path4
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

.....same as the above

 paths:
        - D:\logs\*\path27\S.Coordinator_Z.*.log*
        - C:\logs\*\path27\S.Coordinator_Z*.log*
      input_type: log
      document_type: path27
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path28\d_*_Tr_app.log.txt
        - C:\logs\*\path28\d_*_Tr_app.log.txt
      input_type: log
      document_type: path28
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\R1_Output\R*\pid_*_rr_*
      input_type: log
      document_type: path29
      multiline.pattern: '<\?xml version="1\.0" encoding="UTF-8"\?>'
      multiline.negate: true
      multiline.match: after  
    -
      paths:
        - D:\logs\*\R2_Output\R*\pid_*_rr_*
      input_type: log
      document_type: path30
      multiline.pattern: '<\?xml version="1\.0" encoding="UTF-8"\?>'
      multiline.negate: true
      multiline.match: after

      registry_file: "C:/ProgramData/filebeat/registry"

Answer 1

After a long investigation when i tried to find a similar problem to what i had with a solution , and after trying my luck in the dicuss elastic forum. I managed to solve this issue.

Since I didn't see this option in the web i am putting it here.

Filebeat harvesting system apparently has it limit when it comes with dealing with a big scale number of open files in the same time. (a known problem and elastic team also provides bunch of config options to help dealing this issue and costume ELK to your need, eg config_options ). I managed to solve my problem with opening 2 more Filebeat services which i configures their prospectors in the following way(an example of A same goes to B):

paths:
    - D:\logs\*\pid_*_rr_*
  input_type: log
  document_type: A 
  multiline.pattern: '<\?xml version="1\.0" encoding="UTF-8"\?>'
  multiline.negate: true
  multiline.match: after
  close_eof: true

In this way, because the services working interdependently Filebeat keep trying to operate them (and not "stuck" on the first prospectors).

I manage in this way to double my harvesting abilities.

posing up the discussion in Elastic website: the discussion