Select lines between two patterns using variables inside SED command

Question

I'm new to shell scripting. My requirement is to retrieve lines between two pattern, its working fine if I run it from the terminal without using variables inside sed cmd. But the problem arises when I put all those below cmd in a file and tried to execute it.

#!/bin/sh
word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
fileC=`cat test.log`
output=`echo $fileC | sed -e "n/\$word/$upto/p"`
printf '%s\n' "$output"

If I use the below cmd in the terminal it works fine

sed -n '/ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34/,/2017-01-03 23:00/ p' test.log

Please suggest a workaround.

Answer 1

If we put aside for a moment the fact you shouldn't cat a file to a variable and then echo it for sed filtering, the reason why your command is not working is because you're not quoting the file content variable, fileC when echo ing. This will munge together multiple whitespace characters and turn them into a single space. So, you're losing newlines from the file, as well as multiple spaces, tabs, etc.

To fix it, you can write:

fileC=$(cat test.log)
output=$(echo "$fileC" | sed -n "/$word/,/$upto/p")

Note the double-quotes around fileC (and a fixed sed expression, similar to your second example). Without the quotes (try echo $fileC ), your fileC is expanded (with the default IFS ) into a series of words, each being one argument to echo , and echo will just print those words separated with a single space. Additionally, if the file contains some of the globbing characters (like * ), those patterns are also expanded . This is a common bash pitfall .

Much better would be to write it like this:

output=$(sed -n "/$word/,/$upto/p" test.log)

And if your patterns include some of the sed metacharacters, you should really escape them before using with sed , like this:

escape() {
    sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$1";
}
output=$(sed -n "/$(escape "$word")/,/$(escape "$upto")/ p" test.log)

Answer 2

The correct approach will be something like:

word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
awk -v beg="$word" -v end="$upto" '$0==beg{f=1} f{print; if ($0==end) exit}' file

but until we see your sample input and output we can't know for sure what it is you need to match on (full lines, partial lines, all text on one line, etc.) or what you want to print (include delimiters, exclude one, exclude both, etc.).