将标准输出重定向到文件显示错误的内容

Question

I have a named pipe 'timer_fifo', stdout of a python script is redirected to timer_fifo while parallely another shell script process reads from it, processes it then redirects it to another file named 'activity_file'.

Python Script:

import time
while True:
    print(time.strftime('%Y_%m_%d', time.gmtime()))
    time.sleep(1)

Executing above python script:

python above_file_name > timer_fifo

Shell script:

while true
do
    read action <timer_fifo;
    echo $action | stdbuf -o0 -e0 -i0 sed -n "s/^\([^[:space:]].*\)/\1 Cinnamon/p" >> activity_file
done

Note: Above sed command appends 'Cinnamon' and redirects to activity file, so if input is '2017_02_01' then it becomes '2017_02_01 Cinnamon', also it ignores the input stream if it starts with space.

Executing above shell script:

./file_name

for some time the tail of file:

2017_02_01 Cinnamon
2017_02_01 Cinnamon
2017_02_01 Cinnamon
2017_02_01 Cinnamon
2017_02_01 Cinnamon
2017_02_01 Cinnamon
2017_02_01 Cinnamon

But after a while the tail of file:

2017_02_01 Cinnamon
2017_02_01 Cinnamon
2017_02_01 Cinnamon
2017_02_01 Cinnamon
017_02_01 Cinnamon
207_20 Cinnamon
270_1 Cinnamon
210_101_2001_20 Cinnamon
2017_02_0 Cinnamon
270_11207_02_01 Cinnamon
107_20 Cinnamon
0020 Cinnamon

As you see, why is the output messed up? Also I may have executed the above scripts more than once, so will multiple processes writing simultaneously to same file cause that sort output? If so please explain why?

Answer 1

If you're only running one copy of your program, then everything is arriving in the fifo in sequence, and buffering does not affect ordering.

If you're running more than one copy of your program, and they're each doing their own buffering of stdout, then you could get partial lines.

I think that's what you're seeing.

I think the (imperfect) solution in this case is to add a line in your python script between the print() and the sleep() as follows:

sys.stdout.flush()

This will make it very likely that whole lines will get written to the fifo in single, uninterrupted system calls, every time print() / flush() are called. It still won't be certain--you get no guarantees about atomicity. But in practice, short strings like this will go as single system calls.

(You could also try changing sys.stdout 's buffering, or use a runtime flag to python, but I think calling flush() as above is the best approach.)

from sys.stdout docs : When interactive, standard streams are line-buffered. Otherwise, they are block-buffered like regular text files.