Bash：从文件中读取正则表达式并将它们代入 sed 作为变量内联

Question

I am stuck with how sed interacts with variables. I am reading a list of regex from a file then substitute it into SED to mask certain sensitive information within a log file. if I hard coded the regex, the SED work perfectly, however it behave differently when used with variable.

con-list.txt contain below:
(HTTP\/)(.{2})(.*?)(.{2})(group\.com)
(end\sretrieve\sfacility\s)(.{2})(.*?)(.{3})$

Not sure if the dollar sign for regex is interfering with the SED command.

input="/c/Users/con-list.txt"
inputfiles="/c/Users/test.log"
echo $inputfiles
while IFS= read -r var
do
  #echo "Searching $var"
  count1=`zgrep -E "$var" "$inputfiles" | wc -l`
  if [ ${count1} -ne 0 ] 
  then
    echo "total:${count1} ::: ${var}"
    sed -r -i "s|'[$]var'|'\1\2XXXX\4\5'|g" $inputfiles #this doesnt work
    sed -r -i "s/(HTTP\/)(.{2})(.*?)(.{2})(group\.com)/'\1\2XXXX\4\5'/g"     $inputfiles #This works
    egrep -in "${var}" $inputfiles
  fi
done < "$input"

I need the SED to accept the regex as variable read from the file. So I could automate masking for sensitive information within logs.

$ ./zgrep2.sh
/c/Users/test.log
total:4 ::: (HTTP\/)(.{2})(.*?)(.{2})(group\.comp\.com\@GROUP\.COM)
sed: -e expression #1, char 30: invalid reference \5 on `s' command's RHS

Answer 1

Your idea was right, but you forgot to leave the regex in the sed command to be under double quotes for $var to be expanded.

Also you don't need to use wc -l to count the match of occurrences. The family of utilities under grep all implement a -c flag that returns a count of matches. That said, you don't even need to count the matches, but use the return code of the command (if the match was found or not) simply as

if zgrep -qE "$var" "$inputfiles" ; then

Assuming you might need the count for debug purposes, you can continue with your approach with modifications to your script done as below

Notice how the var is interpolated in the sed substitution, leaving it expanded under double-quotes and once expanded preserving the literal values using the single-quote.

while IFS= read -r var
do
  count1=$(zgrep -Ec "$var" "$inputfiles")
  if [ "${count1}" -ne 0 ] 
  then
    sed -r -i 's|'"$var"'|\1\2XXXX\4\5|g' "$inputfiles"
    sed -r -i "s/(HTTP\/)(.{2})(.*?)(.{2})(group\.com)/'\1\2XXXX\4\5'/g" "$inputfiles"
    egrep -in "${var}" "$inputfiles"
  fi
done < "$input"

Answer 2

You need:

sed -r -i "s/$var"'/\1\2XXXX\4\5/g' $inputfiles

You also need to provide sample input (a useful bit of the log file) so that we can verify our solutions.

EDIT: a slight change to $var and I think this is what you want:

$ cat ~/tmp/j
Got creds for HTTP/PPCKSAPOD81.group.com
Got creds for HTTP/PPCKSAPOD21.group.com
Got creds for HTTP/PPCKSAPOD91.group.com
Got creds for HTTP/PPCKSWAOD81.group.com
Got creds for HTTP/PPCKSDBOD81.group.com
Got creds for HTTP/PPCKSKAOD81.group.com
$ echo $var
(HTTP\/)(.{2})(.*?)(.{2})(.group\.com)
$ sed -r "s/$var"'/\1\2XXXX\4\5/' ~/tmp/j 
Got creds for HTTP/PPXXXX81.group.com
Got creds for HTTP/PPXXXX21.group.com
Got creds for HTTP/PPXXXX91.group.com
Got creds for HTTP/PPXXXX81.group.com
Got creds for HTTP/PPXXXX81.group.com
Got creds for HTTP/PPXXXX81.group.com