How to grep repeating lines in a file?

Question

I am new to shell scripting.

I have a log file and I want a script to alert me if a line repeats. For example:

I have logs such as below:

2016-04-07 06:46:34,750 INFO: Error detected

2016-04-07 06:46:34,801 INFO: Case cleared

2016-04-07 06:46:35,744 INFO: Error detected

2016-04-07 06:48:34,558 INFO: Error detected

In the above logs, there are two repeated lines with 'Error detected' continuously. I want to get alerted if this happens.

Answer 1

You can use this awk to print duplicated line:

awk -F ': ' 'seen[$NF]++ > 1' file

Output:

2016-04-07 06:48:34,558 INFO: Error detected

Answer 2

$ awk '!NF{next} {orig=$0; $1=$2=""} $0==prev{print orig} {prev=$0}' file
2016-04-07 06:48:34,558 INFO: Error detected

Answer 3

You can try this

awk -v prev="" '
$0 ~ /^$/ {next}
{
    info=$4" "$5;
    if(prev == info)
        printf("repeating line: %s\n",$0);
    prev = info;
}' infile

Output:

repeating line: 2016-04-07 06:48:34,558 INFO: Error detected

Answer 4

This will report only the second consecutive error:

awk '
    /^[[:blank:]]*$/ {next}
    /Error detected$/ {errors++}
    !/Error detected$/ {errors=0}
    errors==2
' file

I'm assuming you would not need to be notified for the 3rd, 27th, etc consecutive errors.