How do I learn how to get quoting right in bash?

Question

I'm constantly confused by the rules for quoting and evaluating when I'm writing bash scripts. I know some of the basics, like the difference between '' and "" and ``, but I still seem to get it wrong far too often, and be reduced to experimenting with trying all sorts of different ways to say the same thing.

Any individual problem I can usually work out by brute force, but I think my conceptual model of how it works must be hopelessly broken in some unknown way.

I have no problem with lisp's quote,eval,read,print,syntax-quote system. In fact I wrote a little kata to help people understand what was going on: http://www.learningclojure.com/2010/11/syntax-quote-kata-for-confused.html

I guess that I'm looking for something similar for bash, (which seems much more complicated). A good model, or a set of exercises that will help me form such a model, that will make me able to look at a complicated shell script where variables are being transformed and evaluated and printed out and read, and work out what's going to happen without having to try it.

Failing that, good methods of debugging the process, and watching what's going on at each stage of the evaluation would be really helpful.

Answer 1

Bruce Barnett's UNIX Shell Quote Tutorial is awesome, and the Bash FAQ / pitfalls / word splitting articles have tons of useful tips. A short summary:

Unquoted strings can contain most characters, but not all (like newlines) , and many of them (including space) will have to be escaped. Just don't use them - If you fall for the temptation you might find that someone who modified the script forgot to include quotes once they became necessary.

Single quoted strings can contain most characters, including NUL and newlines, but not single quotes, so they are also useful only for simple values.

Backticks are for commands. They should only be used if your shell does not support $() . Example:

current_dir=`pwd` # BAD! Don't do this!

That command is bad, because when the right hand side of an assignment is not quoted the shell performs word splitting on it. It often leads to hard-to-reproduce bugs, because whitespace is difficult to check visually. To quote commands you have to use double quotes :

current_dir="$(pwd)" # OK, but loses newlines at EOF

Newlines at EOF are especially tricky. You can add a single character and strip it by using for example

# Works for some commands, but not pwd
current_dirx="$(pwd; echo x)"
current_dir="${current_dirx%x}"
printf %s "$current_dir"

, but there's an additional difficulty because some commands (like pwd ) will add a newline at the end of their output anyway , so you might have to remove that as well:

# Works for some commands, including pwd
current_dirx="$(pwd; echo x)"
current_dir="${current_dirx%$'\nx'}"
printf %s "$current_dir"

Double quotes can contain any character (Try echo -ne "\\0" | wc -c ), but note that variables can't contain the NUL character.

ANSI-C quotes can contain any characters except NUL (Try echo -ne $'\\0' | wc -c ), and provides handy escape codes to make it easier to work with special characters:

printf %s $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'
printf %q $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'
touch -- $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'
rm -- $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'

Answer 2

Use singlequotes '' for quoting raw text (even backslashes do not escape anything in singlequotes):

> echo '\'
\
> echo '$PATH'
$PATH

Use doublequotes "" for quoting text which contains things the shell shall evaluate like variables ( $bla ), subshell calls ( $(ls) ), and evaluations ( $((5 + 3)) ).

> echo "$PATH"
/home/alfe/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> echo "$(ls | tail -1)"
bla
> echo "$((5 + 3))"
8

Use backticks `` if for some reason you cannot use $() (eg in the rare cases where you have to you sh instead of bash . Normally use $() to gather the output of a subshell into the current command.

> echo "$(ls | tail -1) is the last file in the current dir."
bla is the last file in the current dir.

One of the main problems I run into with bash code of other people is missing doublequotes around something which often is just a word but in rare cases may be more than one word, or which can contain special characters. So use doublequotes wherever they are possible.

> a="four    spaces"
> echo $a
four spaces
> echo "$a"
four    spaces

Answer 3

At the shell prompt,

set -x
set -v

and create the following python program 'args.py'

#!/usr/bin/env python

import sys
print sys.argv
for arg in sys.argv[1:]:
    for c in arg:
        print c,"|",
    print

Then experiment with command invocations like:

U=hello\ world ; V="-->$U<--"; W="1 $U 2 $V 3"; args.py $W

Until you realize that there is no logical way of thinking about what is going on. It really is all done by capricious magic shell pixies.