bash & sed: replace multiple strings in variable with for loop

Question

Using bash, I'm trying to replace multiple lines in a stored variable with sed and then print the variable with the line changes.

The list variable looks like this - the virtmx elements will all be unique - non repeating

$list variable:

domain1.com     =====>  dest.domain.com:25
domain2.com     =====>  virtmx-0050:25
domain3.com     =====>  dest.domain2.com:25
domain4.com     =====>  domain.example.com:25
domain5.com     =====>  dest.domain3.com:25
domain6.com     =====>  virtmx-0051:25
domain7.com     =====>  dest.domain4.com:25
domain8.com     =====>  dest.domain5.com:25
domain9.com     =====>  dest.domain6.com:25

where I want to replace all virtmx-[num] with another line from a file on the system.

somefile.log has both line[num] and the replacement line together:

example of lines in file.log:

cat /somedir/somefile.log | grep virtmx-*
<mta id="50" host="virtmx-0050:25" dns_type="MX" desc="mxgroupname">
<mta id="51" host="virtmx-0051:25" dns_type="MX" desc="mxgroupname2">

example of line extraction with awk:

cat /somedir/file.log | grep virtmx-0050 | awk -F "\"" '{print $4,$8}'
virtmx-0050:25 mxgroupname
virtmx-0051:25 mxgroupname2

desired output:

domain1.com     =====>  dest.domain.com:25
domain2.com     =====>  mxgroupname
domain3.com     =====>  dest.domain2.com:25
domain4.com     =====>  domain.example.com:25
domain5.com     =====>  dest.domain3.com:25
domain6.com     =====>  mxgroupname2
domain7.com     =====>  dest.domain4.com:25
domain8.com     =====>  dest.domain5.com:25
domain9.com     =====>  dest.domain6.com:25

$listoflines holds just the lines themselves:

line1
line2

I'm using awk to extract the lines then create a sed replacement string - which I use to replace entries in the $output variable:

line1/replacement1

I think I'm close but the below code is only replacing the first instance in the loop then start over and printing the whole variable again with a new replacement, but missing the first. Instead I want to print the whole variable once with all the replacements. The amount of replacements can vary so that's why I'm trying to use a for loop.

for i in $listoflines
do
    replace=`cat /dir/somemfile.log | grep $i | awk -F "\"" '{print $4,$8}' | sed 's/ /\//'`
    echo "$list" | sed 's/'$replace'/'
done

Answer 1

I'm far from an awk guru, but a humble attempt based on glenn's answer:

awk '
    NR==FNR {split($0,x,"\""); r[x[4]]=x[8]; next}
    /virtmx/ {$3=r[$3]}
    {print $1,$2,$3}
' file.log <(echo "$list")

output:

domain1.com =====> dest.domain.com:25
domain2.com =====> mxgroupname
domain3.com =====> dest.domain2.com:25
domain4.com =====> domain.example.com:25
domain5.com =====> dest.domain3.com:25
domain6.com =====> mxgroupname2
domain7.com =====> dest.domain4.com:25
domain8.com =====> dest.domain5.com:25
domain9.com =====> dest.domain6.com:25

discussion:

• Sending two 'files' to awk, file.log and <(echo "$list") .

• NR==FNR is used to process only the first file

  1. split file.log on " and assign an associative array r with pattern->replacements.
  2. process $list as a file, replace using the assoc array if line matches virtmx , and last print the three columns.

Edit: Removed NR!=FNR by using next in the first line

Answer 2

What you should do:

• in awk, instead of creating a sed string "line1/replacement", create an associative array repl[line1] = replacement
• then, still in awk, process the $output (use a process substitution to use a file-like thingy), and use if ($NF in repl) $NF = repl[$NF]; print if ($NF in repl) $NF = repl[$NF]; print

I'd need to see your existing code to give you more concrete advice.