Bash Script Looping over line input

Question

I'm doing the following, which basically works. The script tries to insert some lines into a file to rewrite it.

But it is stripping all blank lines and also all line padding. The main problem is that it does not process the last line of the file. I'm not sure why.

     while read line; do
        <... process some things ...>
        echo ${line}>> "${ACTION_PATH_IN}.work"
     done < "${ACTION_PATH_IN}"

What can be done to fix this?

Answer 1

while IFS= read -r line; do
    ## some work
    printf '%s\n' "$line" >> output
done < <(printf '%s\n' "$(cat input)")

• An empty IFS tells read not to strip leading and trailing whitespace.
• read -r prevents backslash at EOL from creating a line continuation.
• Double-quote your parameter substitution ( "$line" ) to prevent the shell from doing word splitting and globbing on its value.
• Use printf '%s\\n' instead of echo because it is reliable when processing values like like -e , -n , etc.
• < <(printf '%s\\n' "$(cat input)") is an ugly way of LF terminating the contents of input . Other constructions are possible, depending on your requirements (pipe instead of redirect from process substitution if it is okay that your whole while runs in a subshell).
  It might be better if you just ensured that it was LF-terminated before processing it.

Answer 2

Best yet, use a tool such as awk instead of the shell's while loop. First, awk is meant for parsing/manipulating files so for a huge file to process, awk has the advantage. Secondly, you won't have to care whether you have the last newline or not (for your case).

Hence the equivalent of your while read loop:

awk '{
  # process lines
  # print line > "newfile.txt"
}' file 

Answer 3

One possible reason for not reading the last line is that the file does not end with a newline. On the whole, I'd expect it to work even so, but that could be why.

On MacOS X (10.7.1), I got this output, which is the behaviour you are seeing:

$ /bin/echo -n Hi
Hi$ /bin/echo -n Hi > x
$ while read line; do echo $line; done < x
$

The obvious fix is to ensure that the file ends with a newline.

Answer 4

First thing, use

echo "$line" >> ...

Note the quotes. If you don't put them, the shell itself will remove the padding.

As for the last line, it is strange. It may have to do with whether the last line of the file is terminated by a \\n or not (it is a good practice to do so, and almost any editor will do that for you).