How to get “instant" output of “tail -f” as input?

Question

I want to monitor a log file, when a new log message match my defined pattern (say contain “error”), then send out an email to me.

To do that, I wrote a python script monitor.py, the main part looks like:

import sys

for line in sys.stdin:
    if "error" in line:
        print line 

It works well when I use tail my.log | python monitor.py tail my.log | python monitor.py , then I switch to tail -f my.log | python monitor.py tail -f my.log | python monitor.py , then it doesn't work, at least not immediately.

I have done some tests, when the new content to the log accumulate up to 8KB, then my python script can get output from tail. So I highly suspect that this is controlled by the stdin/stdout buffer size. How can I get the output immediately?

One more question, when I use tail -f my.log and tail -f my.log | grep error tail -f my.log | grep error , why it could show me the output immediately?

Answer 1

Most Linux programs will use line buffering if stdout is connecting to a TTY and full buffering otherwise. You can use stdbuf to force line buffering.

stdbuf -oL tail -f my.log | python monitor.py

Answer 2

There's a patch to add unbuffered output to tail , dating from 2008. which appears to have been rejected and my own (BSD) manpage does not indicate it. Perhaps you could download coreutils , apply the patch, compile tail yourself and it may still work?