使用Elastic / Kibana搜索日志条目频率的模式？

Question

I need to take millions of entries from a log (which span the past couple of years), and somehow, using the timestamp field, determine which periods, using days of the week as the grouping criteria, show the least activity. The goal is to show that, for example, Wednesdays between 02:00 and 04:00 has historically shown the lowest level of activity. So, I'm imagining a graph, with time periods on the X-axis (00:00 - 00:14, 00:15 - 00:29, 00:30 - 00:44... or similar... you get the idea) and some kind of representation of log activity on the Y-axis. It would show 7 graph lines, one for each day of the week. This would make it trivial to determine from the graph which period is quietest.

I've not personally used Kibana before, but from what I know about it, it seems likely that this is the best tool to use for this kind of task.

Is there a feature or plugin, or something that has this ability already? Or will I be needing to develop a custom solution to this?

Answer 1

In the end, I gave up on Kibana/Elastic. There's probably a way of doing it, but instead, I just used MySQL:

SELECT 
    t.bucket,
    COALESCE(SUM(total), 0) AS total
FROM
    tmp_time_bucket t
        LEFT JOIN
    (SELECT 
        DATE_FORMAT(FROM_UNIXTIME(FLOOR((UNIX_TIMESTAMP(launchtime)) / 300) * 300), '%H:%i:00') AS bucket,
            COUNT(launchtime) AS total
    FROM
        launchjobs
    WHERE
        launchtime <> '0000-00-00 00:00:00'
            AND DAYNAME(launchtime) = 'wednesday'
    GROUP BY FROM_UNIXTIME(FLOOR((UNIX_TIMESTAMP(launchtime)) / 300) * 300)
    ORDER BY launchtime ASC) m ON t.bucket = m.bucket
GROUP BY bucket
ORDER BY bucket ASC

...where tmp_time_bucket is a table with a single VARCHAR(8) column named bucket which contains all 288 5 minute time buckets in a 24 hour period (so, "00:00:00", "00:05:00", ... "23:50:00", "23:55:00" , you get the idea)

I ran this 7 times, once for each day, and exported the resultset to CSV. Then I used http://plot.ly and imported the data and made the graph that I needed, which (if you're interested) can be seen here: https://plot.ly/~theplankmeister/7/?share_key=FZERWAphDIQsa1swGtixb7

Looking at the graph, I can easily see that the answer I was looking for in the data is Thursday 22:45 until Friday 00:55.

Hope this helps someone in the future!