How to push logs to elasticsearch in filebeat instantly?

Question

hear is my filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - ../typescript/rate-limit-test/logs/*.log
  json.message_key: "message"
  json.keys_under_root: true
  json.overwrite_keys: true
  scan_frequency: 1s

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

logging.level: debug

output.elasticsearch:
  hosts: ["34.97.108.113:9200"]
  index: "filebeat-%{+yyyy-MM-dd}"
setup.template:
  name: 'filebeat'
  pattern: 'filebeat-*'
  enabled: true
setup.template.overwrite: true
setup.template.append_fields:
- name: time
  type: date

processors:
  - drop_fields:
      fields: ["agent","host","ecs","input","log"]

setup.ilm.enabled: false`

I changed scan_frequncy but elasticsearch couldn't get logs faster

How can i get logs in elasticsearch instantly?

Please help me..

Answer 1

There will be never an 'instantly' available logline in elasticsearch. The file needs to be watched for a considerable amount of changes or time, then the newly added lines need to be sent to elasticsearch in a bulk request and indexed into the appropriate shard on the correct cluster node. Network latency, TLS, authentification + authorization, concurrent write/search load: all the things affects the 'instantly' experience.

The speed of log ingestion and NRT (near-real-time search) depends on many factors and configuration options in elasticsearch and filebeat.

Regarding tuning elasticsearch for indexing speed, have a look at this documentation , and apply what you have missed yet. A brief overview:

• Disable swapping and enable memory locking ( bootstrap.memory_lock: true )
• Consider reducing index.refresh_interval (defaults to 1s) for the index in order to have the docs flushed more often (produces more IO in the cluster)

For Filebeat, there is also good documentation about tuning , but in general, I see the following options:

• Try different output.elasticsarch.bulk_max_size values (defaults to a batch size of 50) and monitor the ingestion speed. For each cluster configuration, there are different optimal settings.
• In high load scenarios, when the logs are written fast, consider increasing the number of workers output.elasticsarch.workers (defaults to 1)
• In the opposite scenario, having just a few log lines being written, consider increasing the close_inactive and scan_frequency value for the harvester. Specifying a more suitable backoff will have an effect on how aggressively Filebeat checks files for updates.