在多个日志文件中查找模式？

Question

I have eight log files in the following format:

log01:

[Tue Feb 24 07:39:37 2015] *** MARK ***
[Tue Feb 24 07:40:38 2015] *** MARK ***
[Wed Feb 25 17:13:33 2015] *** MARK ***
[Wed Feb 25 17:14:09 2015] *** MARK ***
[Wed Feb 25 17:16:46 2015] *** MARK ***
[Wed Feb 25 17:17:48 2015] *** MARK ***
[Wed Feb 25 17:22:31 2015] *** MARK ***
[Wed Feb 25 19:10:36 2015] *** MARK ***
[Wed Feb 25 19:10:52 2015] *** MARK ***
[Wed Feb 25 19:11:08 2015] *** MARK ***
[Wed Feb 25 19:11:34 2015] *** MARK ***
[Wed Feb 25 19:12:00 2015] *** MARK ***
[Wed Feb 25 19:12:26 2015] *** MARK ***
[Wed Feb 25 19:13:17 2015] *** MARK ***
[Wed Feb 25 19:13:33 2015] *** MARK ***
[Wed Feb 25 19:15:05 2015] *** MARK ***
[Wed Feb 25 19:37:53 2015] *** MARK ***
[Wed Feb 25 19:38:19 2015] *** MARK ***
[Wed Feb 25 19:38:35 2015] *** MARK ***
[Wed Feb 25 23:08:47 2015] *** MARK ***
[Wed Feb 25 23:09:28 2015] *** MARK ***
[Wed Feb 25 23:11:55 2015] *** MARK ***
[Wed Feb 25 23:12:21 2015] *** MARK ***
[Wed Feb 25 23:12:52 2015] *** MARK ***
[Wed Feb 25 23:13:08 2015] *** MARK ***
...

log02:

[Wed Feb 25 07:01:39 2015] *** MARK ***
[Wed Feb 25 17:13:49 2015] *** MARK ***
[Wed Feb 25 17:15:20 2015] *** MARK ***
[Wed Feb 25 17:16:47 2015] *** MARK ***
[Wed Feb 25 17:17:38 2015] *** MARK ***
[Wed Feb 25 17:19:56 2015] *** MARK ***
[Wed Feb 25 17:22:53 2015] *** MARK ***
[Wed Feb 25 19:10:47 2015] *** MARK ***
[Wed Feb 25 19:11:13 2015] *** MARK ***
[Wed Feb 25 19:11:34 2015] *** MARK ***
[Wed Feb 25 19:11:50 2015] *** MARK ***
[Wed Feb 25 19:12:11 2015] *** MARK ***
[Wed Feb 25 19:12:37 2015] *** MARK ***
[Wed Feb 25 19:12:53 2015] *** MARK ***
[Wed Feb 25 19:13:14 2015] *** MARK ***
[Wed Feb 25 19:13:40 2015] *** MARK ***
[Wed Feb 25 19:14:06 2015] *** MARK ***
[Wed Feb 25 19:14:22 2015] *** MARK ***
[Wed Feb 25 19:14:38 2015] *** MARK ***
[Wed Feb 25 19:38:30 2015] *** MARK ***
[Wed Feb 25 21:17:08 2015] *** MARK ***
[Wed Feb 25 23:08:56 2015] *** MARK ***
[Wed Feb 25 23:10:37 2015] *** MARK ***
[Wed Feb 25 23:11:08 2015] *** MARK ***
[Wed Feb 25 23:11:24 2015] *** MARK ***
[Wed Feb 25 23:12:20 2015] *** MARK ***
[Wed Feb 25 23:12:46 2015] *** MARK ***
...

Every log file is generated by an instance of the same program reading different sensors. A log entry is created if a sensor detects an issue. If every sensor detects an issue within about a minute, it indicates a global problem has occured. For example:

The log entries [Tue Feb 24 07:39:37 2015] *** MARK *** and [Tue Feb 24 07:40:38 2015] *** MARK *** from log01 do not corresponds to anything in log02 so this is not a global problem and can be ignored. The log entries [Wed Feb 25 07:01:39 2015] *** MARK *** and [Wed Feb 25 21:17:08 2015] *** MARK *** in log02 can also be ignored.

However, entry [Wed Feb 25 19:10:36 2015] *** MARK *** in log01 and [Wed Feb 25 19:10:47 2015] *** MARK *** in log02 is within a minute so this indicates a global problem that lasts until entry [Wed Feb 25 19:15:05 2015] *** MARK *** in log01 and [Wed Feb 25 19:14:38 2015] *** MARK *** in log02. So I can conclude that from around 19:10 to 19:15 on Feb 25 something was wrong.

I'm looking for suggestions and tips on how to approach this problem, preferably by using UNIX utilities.

Answer 1

You can try something like this:

#!/bin/bash

for n in $(awk -F' ' '{print $4;}' log01 | cut -c1-5)
do
    if (grep -q $n log02)
    then
        echo "Error on $n"
    fi
done

• The command awk -F' ' '{print $4;}' log01 | cut -c1-5 awk -F' ' '{print $4;}' log01 | cut -c1-5 extracts the hour ( hh:mm ) from your log01 file.
• grep -q $n log02 search for this hour and report and error if found it.

Answer 2

I think the elegant way would be to use Perl to read all the files in and create an array of lists. The first array would be indexed by the time rounded to the nearest minute, and you would push a 1 onto the list at a given time if you were reading from file01 and a 2 if you were reading from file02 . Then at the end, you would iterate through the first array looking for lists with length greater than one. A Perl tag might help there.

If you don't like Perl, you could put something clumsier together, like this.

Step 1: Choose a start time earlier than the earliest time you want.

Step 2: Parse each file, outputting one line per minute of input data. That line is either 0 or 1 depending whether there is a problem or not. One line per minute ensures that all the files line up matching the minute across all 8.

Step 3: Use paste to put all the 8 output files together like this:

paste -d, file{1..8}

1,0,1,1,1,0,1,1
1,1,1,1,1,1,1,1
0,0,1,1,0,0,0,1
0,1,1,1,0,1,0,1
0,0,1,1,0,0,0,1
0,1,1,1,0,1,0,1
1,0,1,1,1,0,1,1
1,1,1,1,1,1,1,1

Step 4: Use awk to look for lines that add up to more than 1.