How to replace string with random line from text file multiple times

Question

I have a script that is meant to rotate thru a file and replace the place holder {{su}} with a random line from a file, the place holder appears multiple times in a file and I need to it to be a randomized each time. Currently it replaces every placeholder with the same line.

#!/bin/bash

subject=$(shuf -n1 *.subjects)
    cat tmp.$file | sed -e "s/{{su}}/$subject/" > output.file

Answer 1

The accepted answer has subtle flaws:

• if {{su}} appears several times on the same line, the same replacement is performed for each pattern {{su}} of that line,
• because read isn't used with IFS= and the -r switch, you'll get other nasty surprises: spaces are not going to be necessarily preserved, and you'll get backslash interpretations (but that's easy to fix),
• if the replacement string contains slashes or other funny characters, sed will be confused.

A method that works, but that involves reading the whole file in memory (it's only good for small files where you have a small number of {{su}} ):

#!/bin/bash

file=$(< filename.txt )

while [[ $file = *'{{su}}'* ]]; do
    repl=$(shuf -n1 file.subjects)
    file=${file/'{{su}}'/"$repl"}
done
printf '%s\n' "$file"

For an approach similar to the accepted answer, ie, reading line by line:

#!/bin/bash

while IFS= read -r line; do
    while [[ $line = *'{{su}}'* ]]; do
        repl=$(shuf -n1 file.subjects)
        line=${line/'{{su}}'/"$repl"}
    done
    printf '%s\n' "$line"
done < filename.txt

Now about the way to select a random line: while shuf is fine, it's an external process and since it's going to be invoked many times (in a subshell), you might consider implementing something similar in Bash. If you have a limited amount of lines, you may consider slurping all the lines into an array and selecting randomly an entry from this array:

#!/bin/bash

mapfile -t replacements < file.subjects
nb_repl=${#replacements[@]}

while IFS= read -r line; do
    while [[ $line = *'{{su}}'* ]]; do
        repl=${replacements[RANDOM%nb_repl]}
        line=${line/'{{su}}'/"$repl"}
    done
    printf '%s\n' "$line"
done < filename.txt

This only works if you have a “small” number of lines in file.subjects (by small, understand less than 32767), and if you're not too worried about the distribution obtained by the modulo. There are very simple workarounds to fix this, though.

Note. You're using shuf -n1 *.subjects . It's an error to invoke shuf with several files (at least with my version of shuf ). So if the glob *.subjects expands to more than one file, you'll get an error.

Note. If you don't want to run into an infinite loop, make sure that the replacements don't contain the {{su}} pattern!

Answer 2

you need a loop. start by using wc -l to count the number of lines in tmp.$file . Then loop count times, in each time perform the two lines of shell script you have. So in each loop, you get a new subject and execute a new sed . The trick is to use the address,address format for the sed command to execute the replace against a single line at a time, passing in the loop counter for the address.

so something like (pseudo code here):

$count = $(wc -l tmp.$file)    
$i=1    
cp tmp.$file > output.file    
while $i < $count    
 subject = $(shuf -n1 *.subjects)    
 cat output.file | sed -e "$i,$is/{{su}}/$subject/" > output.file    
 $i=$i+1    
end while

Answer 3

In that case you need to iterate line by line in your file and generate random string every iteration. It will check all lines for {{su}} pattern and if finds it will substitute it with random string from another file:

while read line
do
subject=$(shuf -n1 *.subjects)
sed -e "s/{{su}}/$subject/g" <<< "$line")
done <1.txt