last five minutes Log4j log analysis using Python

Question

I'm trying to analyze my java application logs ( using log4j ) logged within last five minutes. I'm planning to use Python to analyze last five minutes logs ans if there is any warnings, have to send a mail to me.

Can any one suggest me, what is the best method of getting log lines for last five minutes?

I have to use python for this.

log format is as below,

2011-11-14 10:15:46 WARN : sample warning
2011-11-14 10:15:47 WARN : sample warning
2011-11-14 10:15:48 WARN : sample warning
2011-11-14 10:15:49 WARN : sample warning

Answer 1

Since your log-file is sorted chronologically, you will have to parse it completely:

from datetime import datetime, timedelta
now = datetime.now()
lookback = timedelta(minutes=5)
oldest = (now - lookback).strftime('%Y-%m-%d %H:%M:%S')

lines = []

with open('logfile.log', 'r') as f:
    for line in f:
        if line[:19] > oldest:
            lines.append(line)

if lines:
    message = '\n'.join(lines)
    # send message per mail...

Answer 2

Analysing "the last 5 minutes" doesn't really make a lot of sense. If it were me I'd have something running constantly and limit the alerts it emits to every 5minutes.

(assuming you want to stick with log4j writing to a file...) While you could handle the file I/O in your python code, it's probably simpler to use 'tail' to pipe the file into your script. Obviously you'll need to plan for how you deal with rotating log files - presumably you already have some mechanism for this - it needs to be told to tell the python code to reopen its file handles after rotation.

It might be a lot simpler to send the data to your python script using a pipe - and let python handle the rotation .

Whichever, don't forget to add a SIGALRM to force a check event in the absence of any log messages.