How can I have a newline in a string in sh?

Question

This

STR=\"Hello\\nWorld\"
echo $STR

produces as output

Hello\\nWorld

instead of

Hello
Wo         


        

Answer 1

Disclaimer: I first wrote this and then stumbled upon this question. I thought this solution wasn't yet posted, and saw that tlwhitec did post a similar answer. Still I'm posting this because I hope it's a useful and thorough explanation.

Short answer:

This seems quite a portable solution, as it works on quite some shells (see comment).
This way you can get a real newline into a variable.

The benefit of this solution is that you don't have to use newlines in your source code, so you can indent your code any way you want, and the solution still works. This makes it robust. It's also portable.

# Robust way to put a real newline in a variable (bash, dash, ksh, zsh; indentation-resistant).
nl="$(printf '\nq')"
nl=${nl%q}

Longer answer:

Explanation of the above solution:

The newline would normally lost due to command substitution, but to prevent that, we add a 'q' and remove it afterwards. (The reason for the double quotes is explained further below.)

We can prove that the variable contains an actual newline character (0x0A):

printf '%s' "$nl" | hexdump -C
00000000  0a  |.|
00000001

(Note that the '%s' was needed, otherwise printf will translate a literal '\n' string into an actual 0x0A character, meaning we would prove nothing.)

Of course, instead of the solution proposed in this answer, one could use this as well (but...):

nl='
'

... but that's less robust and can be easily damaged by accidentally indenting the code, or by forgetting to dedent it afterwards, which makes it inconvenient to use in (indented) functions, whereas the earlier solution is robust.

Now, as for the double quotes:
The reason for the double quotes " surrounding the command substitution as in nl="$(printf '\nq')" is that you can then even prefix the variable assignment with the local keyword or builtin (such as in functions), and it will still work on all shells, whereas otherwise the dash shell would have trouble, in the sense that dash would otherwise lose the 'q' and you'd end up with an empty 'nl' variable (again, due to command substitution).
That issue is better illustrated with another example:

dash_trouble_example() {
    e=$(echo hello world) # Not using 'local'.
    echo "$e" # Fine. Outputs 'hello world' in all shells.

    local e=$(echo hello world) # But now, when using 'local' without double quotes ...:
    echo "$e" # ... oops, outputs just 'hello' in dash,
              # ... but 'hello world' in bash and zsh.

    local f="$(echo hello world)" # Finally, using 'local' and surrounding with double quotes.
    echo "$f" # Solved. Outputs 'hello world' in dash, zsh, and bash.

    # So back to our newline example, if we want to use 'local', we need
    # double quotes to surround the command substitution:
    # (If we didn't use double quotes here, then in dash the 'nl' variable
    # would be empty.)
    local nl="$(printf '\nq')"
    nl=${nl%q}
}

Practical example of the above solution:

# Parsing lines in a for loop by setting IFS to a real newline character:

nl="$(printf '\nq')"
nl=${nl%q}

IFS=$nl

for i in $(printf '%b' 'this is line 1\nthis is line 2'); do
    echo "i=$i"
done

# Desired output:
# i=this is line 1
# i=this is line 2

# Exercise:
# Try running this example without the IFS=$nl assignment, and predict the outcome.