How to use awk/sed/grep to filter out only the latest string in a real-time log file

Question

this thing is really confusing me. Suppose I have a file like this

09:35:24 03/04/2014  Pink Floyd the best band in the world KING KING
09:44:21 03/04/2014 Led Zeppelin the greatest hard rock band ever 
09:54:21 03/04/2014 Bon scott I love Bon scott KING KING
10:15:23 03/04/2014 AC/DC is the best
10:35:43 03/04/2014 It's all a joke 
12:46:55 03/04/2014 No value nothing is there here KING KING 
15:35:43 03/04/2014 It's all a joke1 
15:39:43 03/04/2014 It's all a joker KING KING
15:55:43 03/04/2014 It's all a jokeeyyyy 

Now the above one is a file which will be created in the morning and it keeps appending till midnight. Now I have to extract lines where the pattern KING KING exists. Mine is a monitoring script so it checks this file every 60s for the pattern KING KING and if it exists I shall put it in a new file for further operations. Now if you see timestamps are different.At 10:15 I got the first 2 lines with the matching pattern. Now at 12:46 when I run the script I don't want the previously found Matching Pattern again I,e at 12:46 I only want "12:46:55 03/04/2014 No value nothing is there here KING KING " and not again the values at 09:54 and 09:35 (I,e Bon scott and Pink Floyd) So basically how do I use grep/sed/awk to filter only the latest and not give me the old ones. Thanks.

EDIT:

at 10:00 My script runs I grep for KING KING and get the Pink Floyd and Bon Scott lines. Now again (say) at 16:00 My script runs I only want the strings

12:46:55 03/04/2014 No value nothing is there here KING KING

and

15:39:43 03/04/2014 It's all a joker KING KING

and not the older ones. Please note there can be many occurrences of KING KING pattern not only 2.

Answer 1

grep "KING KING" <your file> | tail -n 1

You said you only wanted the most recent line containing KING KING. You can use tail -n 1 to grab just the last one, which is the most recent one assuming your file appends, not prepends, lines.

If it prepends you can use head instead.

EDIT:

If you would like to look at the potentially new KING KING lines that may come in as your original log updates, you can use tail -f instead up there.

Otherwise, if you want just the latest one but you want it to update, you're going to have to keep polling it over and over.

Example:

while true
do
    grep "KING KING" <your file> | grep -vf <your temporary file>
    grep "KING KING" <your file> > <your temporary file>
    sleep 2
done

That's just off the top of my head. It might need some tweaks but I don't have access to a terminal right now. Also, you don't need to rewrite temp file every time, and you can use cat to just append to the end of your temp file but I'll leave it to you to work that one out though. :)

Answer 2

You could keep a temporary file with the day's previous matches in:

touch matches.txt
grep "KING KING" output.log | grep -vFf matches.txt | tee -a matches.txt

This will filter out any previous matches stored in matches.txt , and also appends the new matches to that file. You will need to empty this at midnight, presumably.