"优化多个 sed 替换的 shell 脚本"

Question

I have a file containing a list of replacement pairs (about 100 of them) which are used by sed to replace strings in files.

The pairs go like:

old|new
tobereplaced|replacement
(stuffiwant).*(too)|\1\2

and my current code is:

cat replacement_list | while read i
do
    old=$(echo "$i" | awk -F'|' '{print $1}')    #due to the need for extended regex
    new=$(echo "$i" | awk -F'|' '{print $2}')
    sed -r "s/`echo "$old"`/`echo "$new"`/g" -i file
done

I cannot help but think that there is a more optimal way of performing the replacements. I tried turning the loop around to run through lines of the file first but that turned out to be much more expensive.

Are there any other ways of speeding up this script?

EDIT

Thanks for all the quick responses. Let me try out the various suggestions before choosing an answer.

One thing to clear up: I also need subexpressions/groups functionality. For example, one replacement I might need is:

([0-9])U|\10  #the extra brackets and escapes were required for my original code

Some details on the improvements (to be updated):

• Method: processing time
• Original script: 0.85s
• cut instead of awk : 0.71s
• anubhava's method: 0.18s
• chthonicdaemon's method: 0.01s

Answer 1

您可以使用sed生成正确格式的sed输入：

sed -e 's/^/s|/; s/$/|g/' replacement_list | sed -r -f - file

Answer 2

I recently benchmarked various string replacement methods, among them a custom program, sed -e , perl -lnpe and an probably not that widely known MySQL command line utility, replace . replace being optimized for string replacements was almost an order of magnitude faster than sed . The results looked something like this (slowest first):

custom program > sed > LANG=C sed > perl > LANG=C perl > replace

If you want performance, use replace . To have it available on your system, you'll need to install some MySQL distribution, though.

From replace.c :

Replace strings in textfile

This program replaces strings in files or from stdin to stdout. It accepts a list of from-string/to-string pairs and replaces each occurrence of a from-string with the corresponding to-string. The first occurrence of a found string is matched. If there is more than one possibility for the string to replace, longer matches are preferred before shorter matches.

...

The programs make a DFA-state-machine of the strings and the speed isn't dependent on the count of replace-strings (only of the number of replaces). A line is assumed ending with \\n or \\0. There are no limit exept memory on length of strings.

---

More on sed. You can utilize multiple cores with sed, by splitting your replacements into #cpus groups and then pipe them through sed commands, something like this:

$ sed -e 's/A/B/g; ...' file.txt | \
  sed -e 's/B/C/g; ...' | \
  sed -e 's/C/D/g; ...' | \
  sed -e 's/D/E/g; ...' > out

Also, if you use sed or perl and your system has an UTF-8 setup, then it also boosts performance to place a LANG=C in front of the commands:

$ LANG=C sed ...

Answer 3

You can cut down unnecessary awk invocations and use BASH to break name-value pairs:

while IFS='|' read -r old new; do
   # echo "$old :: $new"
   sed -i "s~$old~$new~g" file
done < replacement_list

IFS='|' will give enable read to populate name-value in 2 different shell variables old and new .

This is assuming ~ is not present in your name-value pairs. If that is not the case then feel free to use an alternate sed delimiter.

Answer 4

Here is what I would try:

1. store your sed search-replace pair in a Bash array like ;
2. build your sed command based on this array using parameter expansion
3. run command.

patterns=(
  old new
  tobereplaced replacement
)
pattern_count=${#patterns[*]} # number of pattern
sedArgs=() # will hold the list of sed arguments

for (( i=0 ; i<$pattern_count ; i=i+2 )); do # don't need to loop on the replacement…
  search=${patterns[i]};
  replace=${patterns[i+1]}; # … here we got the replacement part
  sedArgs+=" -e s/$search/$replace/g"
done
sed ${sedArgs[@]} file

This result in this command:

sed -es/old/new/g -es/tobereplaced/replacement/g file

Answer 5

You can try this.

pattern=''
cat replacement_list | while read i
do
    old=$(echo "$i" | awk -F'|' '{print $1}')    #due to the need for extended regex
    new=$(echo "$i" | awk -F'|' '{print $2}')
    pattern=${pattern}"s/${old}/${new}/g;"
done
sed -r ${pattern} -i file

This will run the sed command only once on the file with all the replacements. You may also want to replace awk with cut . cut may be more optimized then awk , though I am not sure about that.

old=`echo $i | cut -d"|" -f1`
new=`echo $i | cut -d"|" -f2`

Answer 6

You might want to do the whole thing in awk:

awk -F\| 'NR==FNR{old[++n]=$1;new[n]=$2;next}{for(i=1;i<=n;++i)gsub(old[i],new[i])}1' replacement_list file

Build up a list of old and new words from the first file. The next ensures that the rest of the script isn't run on the first file. For the second file, loop through the list of replacements and perform them each one by one. The 1 at the end means that the line is printed.

Answer 7

{ cat replacement_list;echo "-End-"; cat YourFile; } | sed -n '1,/-End-/ s/$/³/;1h;1!H;$ {g
t again
:again
   /^-End-³\n/ {s///;b done
      }
   s/^\([^|]*\)|\([^³]*\)³\(\n\)\(.*\)\1/\1|\2³\3\4\2/
   t again
   s/^[^³]*³\n//
   t again
:done
  p
  }'

More for fun to code via sed. Try maybe for a time perfomance because this start only 1 sed that is recursif.

for posix sed (so --posix with GNU sed)

explaination

• copy replacement list in front of file content with a delimiter (for line with ³ and for list with -End- ) for an easier sed handling (hard to use \\n in class character in posix sed.
• place all line in buffer (add the delimiter of line for replacement list and -End- before)
• if this is -End-³ , remove the line and go to final print
• replace each first pattern (group 1) found in text by second patttern (group 2)
• if found, restart ( t again )
• remove first line
• restart process ( t again ). T is needed because b does not reset the test and next t is always true.

Answer 8

Thanks to @miku above;

I have a 100MB file with a list of 80k replacement-strings.

I tried various combinations of sed's sequentially or parallel, but didn't see throughputs getting shorter than about a 20-hour runtime.

Instead I put my list into a sequence of scripts like "cat in | replace aold anew bold bnew cold cnew ... > out ; rm in ; mv out in".

I randomly picked 1000 replacements per file, so it all went like this:

# first, split my replace-list into manageable chunks (89 files in this case)
split -a 4 -l 1000 80kReplacePairs rep_

# next, make a 'replace' script out of each chunk
for F in rep_* ; do \
    echo "create and make executable a scriptfile" ; \
    echo '#!/bin/sh' > run_$F.sh ; chmod +x run_$F.sh ; \
    echo "for each chunk-file line, strip line-ends," ; \
    echo "then with sed, turn '{long list}' into 'cat in | {long list}' > out" ; \
    cat $F | tr '\n' ' ' | sed 's/^/cat in | replace /;s/$/ > out/' >> run_$F.sh ;
    echo "and append commands to switch in and out files, for next script" ; \
    echo -e " && \\\\ \nrm in && mv out in\n" >> run_$F.sh ; \
done

# put all the replace-scripts in sequence into a main script
ls ./run_rep_aa* > allrun.sh

# make it executable
chmod +x allrun.sh 

# run it
nohup ./allrun.sh &

.. which ran in under 5 mins, a lot less than 20 hours !

Looking back, I could have used more pairs per script, by finding how many lines would make up the limit.

xargs --show-limits </dev/null 2>&1 | grep --color=always "actually use:"
    Maximum length of command we could actually use: 2090490

So just under 2MB; how many pairs would that be for my script ?

head -c 2090490 80kReplacePairs | wc -l

    76923

So it seems I could have used 2 * 40000-line chunks

Answer 9

to expand on chthonicdaemon<\/code> 's solution

<replacement_list perl -p -0 -e '
  s/\//\\\//g;
  s/([^\n]+)\n([^\n]+)(?:\n([^\n]+)(?:\n([^\n]+))?)?/s\/\1\/\2\/\3;/g
'