在Linux中快速查找巨大文件中最后N次出现的字符串

Question

I'm working with an application that generates gigantic log files (2.5GB per day). Occasionally, I need to gather info about the state of the app by searching through the log for select strings.

This is running on a small CentOS Linux system and since it's a production environment I want to minimize the CPU load of this type of search.

What is the most efficient way to find the last 50 occurrences of a string in a large file? The fastest I was able to come up with is this:

tac file.log | grep 'some string' -m50 | tac

Is that as fast as I'm going to get or are there better options?

Also, WHY is this fast? I expected the "tac" to reverse the whole file resulting in slower performance, but that does not appear to be the case.

Update:

An example scenario: say the application logs statistics about its memory utilization every 5 minutes. If I wanted to see the trends over the past hour, I would currently do something like this:

tac file.log | grep 'Memory' -m12 | tac

Answer 1

What you have is good. The reason tac isn't slow is that it doesn't need to read the whole file and reverse it. Instead, it can seek to the last byte of the file and read backward from there. And once your grep finds enough lines, it will stop, SIGPIPE will be raised in the first tac , and the remainder of the input file need not be read at all.