Bash Terminal: Write only specific lines to logfile

Question

I'm running simulation with lots of terminal output, which would exceed my disc space, if I'd save it to a logfile (eg by "cmd > logfile"). Now I would like to follow the entire terminal output, but at the same time I would like to save specific data/lines/values from this output to a file.

1) Is there a way to that in bash?

2) Or as alternative: Is it possible to save the logfile, extract the needed data and then delete the processed lines to avoid generating a huge logfile?

Answer 1

If you want to save into logfile only the output containing mypattern but you want the see all the output at the terminal, you could issue:

cmd 2>&1 | tee /dev/tty | grep 'mypattern' > logfile

I also assumed that the output of cmd may be directed to the standard output stream as well as to the standard error stream, by adding 2>&1 after cmd .

Answer 2

What criteria are you using to decide which lines to keep?

1

One common filter is to just store stderr.

cmdlist 2>logfile # stdout still to console

2

For a more sophisticated filter, if you have specific patterns you want to save to the log, you can use sed . Here's a simplistic example -

seq 1 100 | sed '/.*[37]$/w tmplog'

This will generate numbers from 1 to 100 and send them all to the console, but capture all numbers that end with 3 or 7 to tmplog . It can also accept more complex lists of commands to help you be more comprehensive -

seq 1 100 | sed '/.*[37]$/w 37.log
                 /^2/w 37.log'

c.f. the sed manual for more detailed breakdowns.

You probably also want error output, so it might be a good idea to save that too.

seq 1 100 2>errlog | sed '/.*[37]$/w patlog'

3

For a more complex space-saving plan, create a named pipe, and compress the log from that in a background process.

$: mkfifo transfer               # creates a queue file on disk that 
$: gzip < transfer > log.gz &    # reads from FIFO in bg, compresses to log
$: seq 1 100 2>&1 | tee transfer # tee writes one copy to stdout, one to file

This will show all the output as it comes, but also duplicate a copy to the named pipe; gzip will read it from the named pipe and compress it.

3b

You could replace the tee with the sed for double-dipping space reduction if required -

$: mkfifo transfer               
$: gzip < transfer > log.gz &
$: seq 1 100 2>&1 | sed '/.*[37]$/w transfer'

I don't really recommend this, as you might filter out something you didn't realize you would need.