How does/frequent unix tee command write stdout terminal output to file? if the output is too big

Question

I am redirecting some tool stdout to tee command so that current progress can be seen on terminal as well as in the log file

Here is the code snippet where I am running tool and its stdout is fed to tee command and this code snippet is written from tcl script.

$(EH_SUBMIT) $(ICC_EXEC) $(OPTIONS) -f ./scripts/$@.tcl | tee -i ./logs/$@.log

I can see current real time progress on the terminal but the same observation is not seen in the log file! and it writes stdout to log file chunk by chunk

How does tee work? Does it write by blocks or time or both? If block what is the minimum block size? If it is time what is minimum duration?

I need to parse real time log entries for some data analytics(as I read log file via tail -f and then push new data as the log file grows).

Answer 1

Unless programs handle buffering on their own, buffering of IO streams is handled in the libc. The standard behaviour is: Buffer output line wise if it goes to a terminal, buffer output block wise if it goes to a non-terminal, meaning a file or a pipe. That's why the output appears in log file as you described it: chunk by chunk. This behaviour is for performance optimization.

On Linux the stdbuf command can be used to run a program with adjusted buffers. You need to run your program like this:

stdbuf -oL your_program >> your.log &
tail -f your.log

-oL means buffer stdout linewise.

Answer 2

From the POSIX spec for tee , emphasis added:

The tee utility shall copy standard input to standard output, making a copy in zero or more files. The tee utility shall not buffer output.

So -- tee isn't your problem. Almost certainly your program is buffering what it's writing to stdout (which is default in many programming languages, including C, when stdout is not to a TTY).

stdbuf -oL yourProgram | tee file

...will, if your program is relying on the standard C library to determine its default behavior, suppress that.