Redirecting tail output into a program

Question

I want to send a program the most recent lines from a text file using tail as stdin.

First, I echo to the program some input that will be the same every time, then send in tail input from an inputfile which should first be processed through sed. The following is the command line that I expect to work. But when the program runs it only receives the echo input, not the tail input.

(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program

However, the following works exactly as expected, printing everything out to the terminal:

echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat

So I tried with another type of output, and again while the echoed text posted, the tail text does not appear anywhere:

(echo "new"  && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') | tee out.txt 

This made me think it is a problem with buffering, but I tried the unbuffer program and all other advice here ( https://superuser.com/questions/59497/writing-tail-f-output-to-another-file ) without results. Where is the tail output going and how can I get it to go into my program as expected?

Answer 1

The buffering problem was resolved when I prefixed the sed command with the following:

stdbuf -i0 -o0 -e0 

Much more preferable to using unbuffer, which didn't even work for me. Dave M's suggestion of using sed's relatively new -u also seems to do the trick.

Answer 2

One thing you may be getting confused by -- | (pipeline) is higher precedence than && (consecutive execution). So when you say

(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program

that is equivalent to

(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') && cat) | ./program

So the cat isn't really doing anything, and the sed output is probably buffered a bit. You can try using the -u option to sed to get it to use unbuffered output:

(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -u -r 'some regex')) | ./program

I believe some versions of sed default to -u when the output is a terminal and not when it is a pipe, so that may be the source of the difference you're seeing.

Answer 3

您可以在sed使用i命令（有关详细信息，请参阅联机帮助页中的命令列表）以在开头执行插入：

tail -f inputfile | sed -e '1inew file' -e 's/this/that/' | ./program