How to ensure consistent newline treatment in bash

Question

I have a simple script that fetches some plain-text file using curl, assigns it to variable; later it echos it and greps it for some lines, putting them into a temp file (I actually want to put them in an array); then iterates thru them and does something. I run it on one machine on Mac OS X and it mostly works. However, at random, either of the following happens:

• curl output is treated as a single line, so at echo/grep stage, the thing breaks;
• entire tmp file is treated as single line when doing echo and for loop over it (I seem to have alleviated this using "read line"), so again it breaks.

Is there any way to ensure that newlines are treated consistently, once and for all? I don't even care how as long as it's always the same, I can also sed results out of one-line curl output if needed.

Updated: from a different place. Logged onto machine, did some stuff, noticed a script doesn't work. Ran some commands (result, abridged, below) to test, logged off, logged back on, behavior changed (back to the expected one). Note the difference between for loop output... Same happens for curl output for me where exact same command on (for all I know) exact same env treats line breaks differently. What I want to do is to run some command at the beginning of every script that makes it 100% sure the behavior never changes.

[root@ip-xxx-xxx-xxx-xxx ~]# cat example.txt 
ip-xxx-xxx-xxx-xxx.ec2.internal
ip-xxx-xxx-xxx-xxx.ec2.internal
[root@ip-xxx-xxx-xxx-xxx ~]# for f in `cat example.txt`; do echo "line $f"; done
line ip-xxx-xxx-xxx-xxx.ec2.internal
ip-xxx-xxx-xxx-xxx.ec2.internal
[root@ip-xxx-xxx-xxx-xxx ~]# while read f; do echo "line $f"; done < "example.txt"
line ip-xxx-xxx-xxx-xxx.ec2.internal
line ip-xxx-xxx-xxx-xxx.ec2.internal
[root@ip-xxx-xxx-xxx-xxx ~]# logout
Connection to ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com closed.
reznor-mbp:trunk2 sergey$ ssh -i ... "root@..."
Last login: ...
[root@ip-xxx-xxx-xxx-xxx ~]# for f in `cat example.txt`; do echo "line $f"; done
line ip-xxx-xxx-xxx-xxx.ec2.internal
line ip-xxx-xxx-xxx-xxx.ec2.internal

Answer 1

First: Don't read lines with for; read them with a while read loop:

while read -r; do
  echo "line $REPLY"
done <example.txt

The problem with for f in $(cat example.txt) is that the output of the cat operation goes through string-splitting and glob expansion before the loop is entered, making behavior dependent on a number of variables (contents of the IFS shell setting, files in the current directory [if potentially glob-expanding names are present], various shell options impacting how globs are interpreted, etc). The while read approach will behave consistently.

Second -- always quote expansions for the same reason: "$line" avoids string-splitting and glob expansion (again, making the above settings and variables moot), whereas bare $line has those occur.

Finally, the example above uses the default REPLY variable because the shell does not trim whitespace characters (that is, characters found in $IFS ) on read under those conditions. This is usually unnecessary (and the same effect can be had by explicitly clearing IFS, as in while IFS= read -r line; do ... ), but since the goal is to be reproducible, we're doing it right. :)

Follow these practices, and you will see consistent behavior between all shells compliant with POSIX sh. (The only major shell which does not comply with POSIX sh -- breaking compatibility by default in places where it considers the standard-mandated behavior unreasonable -- is zsh).

Answer 2

You mention echo . Do you write

echo $multiLineVar

or

echo "$multiLineVar"

In the first case, the shell will treat embedded newlines as white-space that separates words, and so echo never sees them.

Answer 3

Simple way is to use a while loop as below

filename=/path/to/file/file.txt
while read variablename

do echo "$variablename" done < "filename" What it does is it reads whole line until \\n