Logstash - ElasticSearch - Kibana :: delay of 10 or more seconds

Question

I've a default "elasticsearch" (ubuntu 14.04, ES v1.2) stack deployed, from redis to Kibana.

I'm sending a @now value with the current date when it's sent, and Elasticsearch assigns his own @timestamp.

Well, if you calculate (@timestamp - @now) there will be more than 10 seconds (sometimes even a minute) of lag/delay.

Is it the normal behaviour? I haven't tune too much my instance, but I'm sending very few events and doesn't look likes a problem of performance/memory/IO.

Any hint is welcome.

Answer 1

You have at least 5 pieces of software along the way (you don't mention what shipper you're using).

First, make sure everything's "warm" when you're looking at results. logstash and elasticsearch are JVM-based, so there's all that overhead to worry about. I usually give them 2 minutes before I start measuring anything.

Secondly, look for buffer sizes, which could cause more impact in a low-volume environment like yours. Does your shipper send every message, or batch (logstash has a default of 50 documents per batch when used as a shipper to redis)? What about when reading from redis (default is 1, but can be changed)? What about sending from logstash to elasticsearch (default is 1,000 though it is also flushed every second)?

What about your hardware all along the chain? CPU utilization? RAM allocation? SSD vs. spinning disks? Network latency? Garbage collection?

How much filtering are you doing on the shipper or the indexer? A lot of bad regexps?

Or even the basics - are the clocks set identically?

[ I can already see the SO police suggesting that this should be a comment and not an answer. However, you'll notice specific things mentioned for the OP to research, and the lesson that there are a lot of knobs to turn. ]