sed param substitution in bash

Question

"I am new at bash development and using sed. I have tried working on this for a couple of days and read and googled as many books on "sed" as I can.

Here is my problem. I need to do a replacement in a line with the "value" of the term inside the pattern I am replacing. I am using back referencing in my regex and can get the term inside the pattern to replace the pattern. But would like to get the "value" of that term.

Below is the code I've gotten to work so far, but can't get it to do that last step.

1 #!/bin/bash
2 
3 val='<%= @test %> this is my <%= @new %> here'
4 echo "Echo: $val"
5 
6 export test='cool value'
7 export new='new value'
8 
9 echo $val | sed -e "s:\<\%\= \@\([A-Za-z0-0]*\) \%\>:\1:g"

For ease of understanding of the "pattern", I have used ":" as the delimiter in the sed "s" statement. What some of the google posts I have seen suggest that to get the value of the back reference term, I should use a $\\1 but that does not seem to work.

The output I am looking for is:

"cool value this is my new value"

The entire pattern is what I want to replace. I'm sorry I was not as explicit as I needed to be in my original post.

The above is just a test to see if I can make the replacement. The eventual purpose will to be able to replace the '<%= @term %>' in configuration files with environment variables (line by line) in configuration files.

Any help would be appreciated.

Answer 1

you add too many slashes. adding them won't make you safer. for instance, \\< and \\> have special meaning, which represent start and end of a word, respectively.

to achieve this, it's much simpler than you think:

sed -e "s/<%= @\([A-Za-z0-9]*\) %>/\1/g"

this should be able to capture both test and new .

to manage sed, you don't need to read a lot too. this tutorial is supposed to be enough: http://www.grymoire.com/Unix/sed.html .

Answer 2

Answer for Revised Question

sed is not the right tool for this. You can substitute the values of shell variables into a sed script before you run the script. But, sed has no way of dynamically looking the value of a shell variable after the sed script has started to run.

shell will do what you want:

#!/bin/bash

val='<%= @test %> this is my <%= @new %> here'

test='cool value'
new='new value'

for var in test new
do
    val=${val//<%= @$var %>/${!var}}
done
echo $val

The above produces the output:

cool value this is my new value here

Unless there is a compelling reason to use shell variables, you may want to consider using bash associative arrays. See below for an example applied to the original question.

Answer for Original Question

What you want is not a good match to sed's capabilities. gawk offers access to shell environment variables and it could be used. It is, however, relatively simple to use bash's built-in parameter expansion capabilities to accomplish what you want:

#!/bin/bash

val='<%= @test %> this is my <%= @new %> here'
echo "Echo: $val"

export test='cool value'
export new='new value'

val=${val//@test/$test}
val=${val//@new/$new}

echo $val

The above produces the output:

Echo: <%= @test %> this is my <%= @new %> here
<%= cool value %> this is my <%= new value %> here

Alternative

If there are many variables like test or new , it might be easier to create an associative array to hold them:

#!/bin/bash

val='<%= @test %> this is my <%= @new %> here'
echo "Echo: $val"

declare -A arr=([test]='cool value' [new]='new value')

for key in "${!arr[@]}"
do
    val=${val//@$key/${arr[$key]}}
done

echo $val

The above produces the output:

Echo: <%= @test %> this is my <%= @new %> here
<%= cool value %> this is my <%= new value %> here

Answer 3

Thanks for the help from @HuStmpHrr and @John1024. @HuStmpHrr was right about the abuse of slashes... and @John1024 seems to be right about the fact that "sed" is not the right tool for the job.

I found "perl" worked the way I wanted (a co-worker figured this out). I would rather not have used perl (just because I'm a anti-perl bigot from the days when I wrote too much bad perl code), but I couldn't figure out how to get the environmental variables to substitute for the ENTIRE pattern.

I came up with the following code that creates the correct output. If anyone else can figure out how to do this in sed or awk, I would appreciate it.

1 #!/bin/bash
2 
3 pat="<%= *@([a-zA-Z0-9_]*) *%>"
4 val='<%= @test %> this is my <%= @new %> here'
5 echo "Echo: $val"
6 
7 export test='cool value'
8 export new='new value'
9 
10 # this is the old attempt with sed. It didn't produce output I wanted
11  echo $val | sed -e 's/<%= @\([A-Za-z0-9]*\) %>/\1/g'
12 
13 # this works the way I want to with Perl
14  echo $val | perl -pe 's/<%= *@([a-zA-Z0-9_]*) *%>/$ENV{$1}/g'

Thanks for the pointers in the right direction.