Trouble with sed: “bad pattern”

Question

I'm working on a script to run on Android (hence the weird shebang path). It involves using sed to comment out certain blocks of code from specified files. Currently, I'm trying to pass an entire sed command to a function, but I'm having a lot of trouble doing so.

This is the script:

#!/system/bin/sh

REM_RCTD=$1
REM_CCMD=$2
REM_TRITON=$3
DEVICE_CODE=$4

COLOR_GRN_PRE="<font color='#00ff00'>"
COLOR_YEL_PRE="<font color='#ffff00'>"
COLOR_POS="</font>"

YELLOW=0
GREEN=1

echoAndExec() {
    CMD="$2"

    if [ "$1" = ${YELLOW} ]; then
        echo "$COLOR_YEL_PRE $CMD $COLOR_POS"
    elif [ "$1" = ${GREEN} ]; then
        echo "$COLOR_GRN_PRE $CMD $COLOR_POS"
    fi

    ${CMD} || exit 1
}

if [ "$REM_RCTD" = "true" ]
then

    CMD="sed -ir -e \_^# LG RCT(Rooting Check Tool)$_,/^$/{/^(#\|$)/!s/^/#/} init.lge.rc"
    echoAndExec ${YELLOW} "${CMD}"
fi

if [ "$REM_CCMD" = "true" ]
then

    CMD="sed -ir -e \_^service ccmd /system/bin/ccmd$_,/^$/{/^(#\|$)/!s/^/#/} init.lge.rc"
    echoAndExec ${YELLOW} "${CMD}"
fi

if [ "$REM_TRITON" = "true" ]
then

    CMD="sed -ir -e /# triton service/,\_chmod 644 /sys/devices/system/cpu/triton/enable_s/^/# / init.${DEVICE_CODE}.power.rc"
    echoAndExec ${YELLOW} "${CMD}"
fi

Every command sent to the echoAndExec() function works fine, except for sed , which returns

sed: bad pattern '\_^#@1 (\)

For the third sed command, it has the same issue with the / .

I've tried a bunch of combinations of quotes, with and without the usage of variables, with different syntax for the pattern beginnings (/#, _), but I'm really lost. As is probably obvious from my script, I'm pretty much a beginner in shell scripting.

Is there maybe a POSIX alternative to sed I could use that might work better? Or is there some combination of quotes and escapes that'll make this work? I just can't figure it out.

If anyone needs any more details, I'll be happy to provide them; I just don't know what would be relevant.

Answer 1

TL;DR: You can use eval in the function and include correct internal quotation marks in the command you pass, which for complex commands is cumbersome and may make code that uses your function very hard to read, but which has some advantages. Or you can pass the commands as multiple arguments to your shell function, after the initial color argument, shift off the color argument after storing or otherwise using it, and run the command as "$@" . See below for full details. (There are also likely methods not covered here at all; perhaps others will write answers about them.)

The Problem

The ${CMD} in ${CMD} || exit 1 ${CMD} || exit 1 in not quoted , so word splitting and globbing are performed. That only does what you actually want in the very simplest of cases. Each of your sed commands contains text that you intend to be passed to sed as a single argument, but which contains spaces. The spaces cause it to split into multiple words, each of which is passed to sed as a separate command-line argument.

Consider the simplified situation where the command you want to run is:

printf '%s\n' 'foo bar' 'baz quux'

When you run printf with %s\\n as its first argument, it prints each of its subsequent arguments on a line by itself. This is a convenient way to check the effects of word splitting and globbing. The output of that particular command is:

foo bar
baz quux

If you assign the whole command to CMD without internal quotes , the simplicity of this particular command makes the problem immediately obvious: there is no way for the shell to know where you do and do not intend for it to perform word splitting.

$ CMD="printf %s\n foo bar baz quux"
$ $CMD
foo
bar
baz
quux

But this is exactly the situation you have with each of your sed commands. Some of the spaces in your sed commands are intended to separate arguments, while others are not, and the shell has no way to know what you want.

Embedding internal quotation marks into the value of CMD will not solve the problem. Not by itself, anyway. Since they are themselves quoted, their special meaning is suppressed. Quote removal does not occur in this situation, so they will simply remain at the edges of the words where you put them. Furthermore, the whitespace you intended for them to quote still causes word splitting:

$ CMD="printf %s\n 'foo bar' 'baz quux'"
$ $CMD
'foo
bar'
'baz
quux'

You can, of course, expand $CMD inside double quotes ( "$CMD" ). But that prevents all word splitting, and attempts to run a program whose name is your entire command, spaces and all. That is not what you want:

$ "$CMD"
printf %s\n 'foo bar' 'baz quux': command not found

There are multiple ways to solve this problem. I'll show two.

Way 1: You can make the function run the command with `eval`

One solution is to expand $CMD in double quotes to prevent word splitting, but instead of running that, pass it as an argument to the eval shell builtin. That causes the shell to parse the contents of CMD the same way it does when those contents actually appear as a line in a script. Note that, since the actual text stored in CMD is not quoted, you must include internal quotes for everything that requires quoting. In simple cases, you can get away with using " " for either the inner quotes or the outer quotes:

$ CMD='printf "%s\n" "foo bar" "baz quux"'
$ eval "$CMD"
foo bar
baz quux

$ CMD="printf '%s\n' 'foo bar' 'baz quux'"
$ eval "$CMD"
foo bar
baz quux

However, for commands that contain characters like $ that may be treated specially if expanded inside double quotes -- like what you are doing with sed -- it is often necessary to use single quotes for both. To achieve this, you can end quoting for just long enough to write a single quote that is itself quoted with \\ , then resume quoting again. That is, you can write a single quote "inside" single quotes as '\\'' .

$ CMD='printf '\''%s\n'\'' '\''foo bar'\'' '\''baz quux'\'''
$ eval "$CMD"
foo bar
baz quux

This approach has the major disadvantage that it can be hard to quote your commands correctly to pass them to your shell function, and extremely hard for you (or someone else) to verify they are correct by inspection. However, it has the advantage that the function has a string that can be run, exactly as written, as your command. Normally this is unimportant, but in your case it may be important, because you are showing the user what the command was. If you want the user to see a command that can be run, verbatim -- that is, you want to show proper quoting to the user, not just run the command correctly -- then this eval -based approach is probably the simplest way to achieve that.

Here's a modified version of your shell function. It takes the color attribute value that goes in your start tag, rather than a number, as the first argument. I actually suggest you do that, as there's no reason $YELLOW and $GREEN cannot just be #ffff00 and #00ff00 , respectively. But however you choose to write it, this should demonstrate how you can use eval in it.

echoAndExec() {
    printf '<font color='\''%s'\''> %s </font>\n' "$1" "$2"
    eval "$2" || exit 1
}

As you can see, I've also replaced echo with printf , since some echo implementations expand escape sequences, which you probably don't want here. (Some can also treat the first argument you pass as though it were an option, but that's not an issue here, since your first argument starts with < , never - .) Finally, I have made the variables lower-case. I suggest using lower-case names for your shell variables , unless you plan to export them as environment variables . It is common both for environment variables and for variables treated specially by shells (eg, PS1 ) to be named in upper case, and using lower case helps avoid conflicts. You can use upper case if you want to, though.

Here's how you might call that function:

$ green='#00ff00'
$ cmd='printf '\''%s\n'\'' '\''foo bar'\'' '\''baz quux'\'''
$ echoAndExec "$green" "$cmd"
<font color='#00ff00'> printf '%s\n' 'foo bar' 'baz quux' </font>
foo bar
baz quux

Or just:

$ green='#00ff00'
$ echoAndExec "$green" 'printf '\''%s\n'\'' '\''foo bar'\'' '\''baz quux'\'''
<font color='#00ff00'> printf '%s\n' 'foo bar' 'baz quux' </font>
foo bar
baz quux

Of course, whether or not you assign the command to a variable first, you do not have to redefine green each time.

Way 2: You can pass the command as multiple arguments to the function

Normally, when you perform parameter expansion in double quotes , word splitting is suppressed entirely. However, the @ parameter is special. * and @ both contain the text of all your positional parameters , one after the other, but they behave differently when expanded in double quotes. With "$*" , you get no word splitting--the positional parameters are joined, with single spaces (or whatever is the first character of $IFS ) between them.

In contrast, with "$@" , word splitting is performed between the positional parameters, but not within them. This is to say that each positional parameter becomes its own word, but a positional parameter that contains whitespace still won't be split any further (as it would be with $* or $@ outside " " ).

This provides exactly the functionality you need to pass a command to your function, in a way that allows you to write the command in a readable manner , and have the function run it correctly. Here's how you might write your function, if you want to take this approach:

echoAndExec() {
    printf '<font color='\''%s'\''> ' "$1"
    shift
    printf '%s </font>\n' "$*"
    "$@" || exit 1
}

Running "$@" initially would not do what you want, because it would have included the first positional parameter at the beginning. To remove the first positional parameter while shifting each of the others down by one (or you may prefer to think of it as shifting them left by one), I used the shift builtin.

That code is somewhat less readable than it could be, because I have avoided introducing any variables in the shell function. The reason I have done this is that the declare and local builtins are not actually required by POSIX , and I am not sure if your shell--and other shells on which you might need to run this script--support them. Without them, assigning to a variable in a shell function causes them to be set for the caller as well. For the particular script you've shown, that doesn't seem like it would be a problem, but I don't know if (or how) you might end up extending the script or what variable names you might use in it later.

Some of the changes shown above, compared to your version of the function, are specific neither to this approach of expanding "$@" nor to the preceding approach of using eval . I have explained those changes in the preceding section. You do not have to write your function exactly this way, though you can; the purpose of the above code is to serve as an example.

It's important to remember that you must call this function in a different way from the way you would call the function you wrote originally (and also differently from the eval -based version shown above). Don't store your commands in a variable first. Just pass each word of the command to the shell function:

$ green='#00ff00'
$ echoAndExec "$green" printf '%s\n' 'foo bar' 'baz quux'
<font color='#00ff00'> printf %s\n foo bar baz quux </font>
foo bar
baz quux

You will notice that this is much, much easier to call, because you don't have to use any quoting beyond what you would use just to run the command directly. In fact, you must not use such additional quoting. It is enormously easier to use this function and to understand what you have written and verify that it is correct.

However, this does have the disadvantage that it is no longer trivial to print the command that was passed in with its original quoting. The shell removes those quotes when the function is called. The function does not have access to them.

You might not care about this, but if you do, then you can make your shell function insert quotes around each argument. They won't necessarily be the same quotes that were used originally, and they might not even be correct if the arguments themselves contained single quotes. But they should typically indicate, in a reasonably unambiguous way, what arguments were passed:

echoAndExec() {
    printf '<font color='\''%s'\''> ' "$1"
    shift
    printf \''%s'\'' ' "$@"
    printf '</font>\n'
    "$@" || exit 1
}

You would use that function the same way. It will look like this:

$ green='#00ff00'
$ echoAndExec "$green" printf '%s\n' 'foo bar' 'baz quux'
<font color='#00ff00'> 'printf' '%s\n' 'foo bar' 'baz quux' </font>
foo bar
baz quux

(As mentioned above, you don't have to redefine green each time. I've just done so, so that it's clear what green means in the versions of the shell function that I have written, and so that it is clear how to test this easily.)

Although you can do that, I underscore that the command it shows will not always be possible to run as shown, because it does not handle internal single quotes correctly. The command shown to the user is pretty good for humans, but not so great for computers. Therefore, although you can use this modified method, it is probably better to go with the eval -based way shown above, if you need to show the user a command with original (or otherwise proper) quoting.

Of course, it is also possible to process the arguments in a more sophisticated way that, for example, properly converts internal occurrences of ' into '\\'' .

Trouble with sed: “bad pattern”

Question

1 answers

solution1
6 ACCPTED 2018-01-06 21:47:42

The Problem

Way 1: You can make the function run the command with `eval`

Way 2: You can pass the command as multiple arguments to the function

Further Reading

Trouble with sed: “bad pattern”

Question

1 answers

solution1 6 ACCPTED 2018-01-06 21:47:42

The Problem

Way 1: You can make the function run the command with eval

Way 2: You can pass the command as multiple arguments to the function

Further Reading

solution1
6 ACCPTED 2018-01-06 21:47:42

Way 1: You can make the function run the command with `eval`