分别重定向和重组stderr / stdout，而不会丢失顺序

Question

I want to execute a command and want to redirect stderr and stdout as below:

stderr and stdout -> should be written ONLY to logs.log file while keeping the order

stderr -> should be printed to SCREEN and also written to errors.log

So far I can redirect them both to screen and the file log.txt like this:

command 2>&1 | tee logs.log

But the above is not what I need.

To make it more clear again what the results needs to be.

After command is executed I need to see on screen only the result of stderr, I need to have one file named errors.log with stderr, and I need to have another file named logs.log with the result of both stdout and stderr in the original order in which they were created.

Answer 1

Preserving perfect order while performing separate redirections is not even theoretically possible without some ugly hackery. Ordering is only preserved in writes (in O_APPEND mode) directly to the same file; as soon as you put something like tee in one process but not the other, ordering guarantees go out the window and can't be retrieved without keeping information about which syscalls were invoked in what order.

So, what would that hackery look like? It might look something like this:

# eat our initialization time *before* we start the background process
sudo sysdig-probe-loader

# now, start monitoring syscalls made by children of this shell that write to fd 1 or 2
# ...funnel content into our logs.log file
sudo sysdig -s 32768 -b -p '%evt.buffer' \
  "proc.apid=$$ and evt.type=write and (fd.num=1 or fd.num=2)" \
  > >(base64 -i -d >logs.log) \
  & sysdig_pid=$!

# Run your-program, with stderr going both to console and to errors.log
./your-program >/dev/null 2> >(tee errors.log)

That said, this remains ugly hackery: It only catches writes direct to FDs 1 and 2, and doesn't track any further redirections that may take place. (This could be improved by performing the writes to FIFOs, and using sysdig to track writes to those FIFOs; that way fdup() and similar operations would work as-expected; but the above suffices to prove the concept).

---

Making Separate Handling Explicit

Here we demonstrate how to use this to colorize only stderr, and leave stdout alone -- by telling sysdig to generate a stream of JSON as output, and then iterating over that:

exec {colorizer_fd}> >(
  jq --unbuffered --arg startColor "$(tput setaf 1)" --arg endColor "$(tput sgr0)" -r '
    if .["fd.filename"] == "stdout" then
      ("STDOUT: " + .["evt.buffer"])
    else
      ("STDERR: " + $startColor + .["evt.buffer"] + $endColor)
    end
  '
)

sudo sysdig -s 32768 -j -p '%fd.filename %evt.buffer' \
  "proc.apid=$$ and evt.type=write and proc.name != jq and (fd.num=1 or fd.num=2)" \
  >&$colorizer_fd \
  & sysdig_pid=$!

# Run your-program, with stdout and stderr going to two separately-named destinations
./your-program >stdout 2>stderr

Because we're keying off the output filenames ( stdout and stderr ), these need to be constant for the above code to work -- any temporary directory desired can be used.

---

Obviously, you shouldn't actually do any of this. Update your program to support whatever logging infrastructure is available in its native language (Log4j in Java, the Python logging module, etc) to allow its logging to be configured explicitly.

Answer 2

This will get you most of the way there:

your_command 2> >(tee -a logs.log errors.log) 1>>logs.log

but I don't think you'll be able to exactly preserve the order of the output in the logs.log file.