从日志文件的 sftp 连接中排除特定行

Question

I have multiple scripts to connect to an sftp server and put/get files. Recently the sftp admins added a large header for all the logins, which has blown up my log files, and I'd like to exclude that as it's blowing up the file sizes as well as making the logs somewhat unreadable.

As is, the commands are of the following format:

sftp user@host <<EOF &>> ${LOGFILE}
get ...
put ...
exit
EOF

Now, I've tried grepping out all the new lines, which all start with a pipe (they basically made a box to put them in).

sftp user@host <<EOF | grep -v '^|' &>> ${LOGFILE}
get ...
put ...
exit
EOF

This excludes the lines from ${LOGFILE} but throws them to stdout instead which means they end up in another log file, where we also don't want them (these scripts are called by a scheduler). Oddly, it also seems to filter out the first line from the connection attempt output.

Connecting to <host>...

and redirect that to stdout as well. Not the end of the world, but I do find it odd.

How can I filter the lines beginning with |so they don't show anywhere?

Answer 1

In

sftp user@host <<EOF &>> ${LOGFILE}

You are redirecting stdout and stderr to the logfile for appending data (&>>). But when you use

sftp user@host <<EOF | grep -v '^|' &>> ${LOGFILE}

you are only redirecting stdout to grep, leaving the stderr output of sftp to pass untouched. Finally, you are redirecting agains stdout and stderr of grep to the logfile.

In fact, you are interested in redirecting both (stdout and stderr) from sftp, so you can use someting like:

sftp user@host <<EOF |& grep -v '^|' >> ${LOGFILE}

or, in older versions of bash, using the specific redirection instead of the shorthand:

sftp user@host <<EOF 2>&1 | grep -v '^|' >> ${LOGFILE}