Sending standard error to standard out not working

Question

I'm having trouble getting the command to send standard error to standard out. I intend to put it in cron, but for now am just running manually.

/home/backups/backup_scripts/backup.sh 2>&1 >> /var/log/backups/backup.log

I was hoping the log file would contain any messages that were printed to standard out, but the log file only showed standard out, and the console output showed the errors.

What am I doing wrong here?

Answer 1

The order of the redirect matters. You should change it to:

/home/backups/backup_scripts/backup.sh >> /var/log/backups/backup.log 2>&1 

This redirects stdout into /var/log/backups/backup.log and then redirects stderr into stdout (which is going into the file).

You can also use the shorter form of:

/home/backups/backup_scripts/backup.sh &>> /var/log/backups/backup.log

&>>file is semantically equivalent to >>file 2>&1

Answer 2

This happened because of the order of the redirections. The 2>&1 means "redirect standard error to the current location of standard out". They are processed left to right so first standard error is redirected to the current standard out (still the terminal) then standard out is redirected to a file. If you switch the order to read

/home/backups/backup_scripts/backup.sh >> /var/log/backups/backup.log 2>&1

it should do what you want. If your shell is bash it has a shortcut to redirect standard out and standard error at the same time: command &> file redirects them both to a file and command &>> file appends both to a file. This order dependency is also mentioned in the bash manual

Note that the order of redirections is significant. For example, the command

 ls > dirlist 2>&1 

directs both standard output and standard error to the file dirlist, while the command

 ls 2>&1 > dirlist 

directs only the standard output to file dirlist, because the standard error was duplicated from the standard output before the standard output was redirected to dirlist.

I think there is some misunderstanding of how file descriptors and redirection works. The kernel has a big table of all the open files on the system that contains the file offset, status flags, and the name of the file itself. Then each process has a table that maps file descriptor numbers to the entries in this global table.

Before any redirections happen the global table might look something like this

A: TERMINAL

and the process table looks like

0: A
1: A
2: A

since descriptor numbers 0, 1, and 2 are stdin, stdout, and stderr respectively.

Then the 2>&1 redirection is processed. This changes the process table so that 2 contains points to the same global entry as 1. However, they are already the same so nothing happens.

When the >> /var/log/backups/backup.log redirection is processed. First the file is opened and assigned to file descriptor 3 in the process table so the tables look like

A: TERMINAL
B: /var/log/backups/backup.log

and

0: A
1: A
2: A
3: B

then standard out is changed to point to the newly opened file (as if 1>&3 were done) so the process table is now

0: A
1: B
2: A
3: B

Note that 2 (stderr) is still pointing to the terminal.

When the redirections are done in the other order, the table after >> /var/log/backups/backup.log looks the same.

0: A
1: B
2: A
3: B

and then the 2>&1 redirection changes what 2 points to giving

0: A
1: B
2: B
3: B

so now stdout and stderr both go to the file.