shell script replace variables in file - error with Sed's -i option for in-place updating

Question

Here is my test.env

RABBITMQ_HOST=127.0.0.1
RABBITMQ_PASS=1234

And I want to use test.sh to replace the value in

Answer 1

ex -sc '%!awk "\
\$1 == \"RABBITMQ_HOST\" && \$2 = \"rabbitmq1\"\
\$1 == \"RABBITMQ_PASS\" && \$2 = 12345\
" FS== OFS==' -cx file

1. POSIX Sed does not support the -i option. However ex can edit files in place

2. Awk is a better tool for this, as the data is separated into records and fields

3. In either case Sed or Awk, you can utilize a newline or ; to do everything in one invocation

4. You have double quoted strings with no variables inside, might as well use single quotes

5. You quoted your file name when it has no characters that need escaping

6. You have several unquoted uses of variables, almost never a good idea

Answer 2

tl;dr:

With BSD Sed, such as also found on macOS, you must use -i '' instead of just -i (for not creating a backup file) to make your commands work; e.g.:

sed -i '' 's/RABBITMQ_HOST=.*/RABBITMQ_HOST='"$RABBITMQ_HOST"'/'  "$Deploy_path"

To make your command work with both GNU and BSD Sed, specify a nonempty option-argument (which creates a backup) and attach it directly to -i:

sed -i'.bak' 's/RABBITMQ_HOST=.*/RABBITMQ_HOST='"$RABBITMQ_HOST"'/'  "$Deploy_path" &&
  rm "$Deploy_path.bak" # remove unneeded backup copy

Background information, (more) portable solutions, and refinement of your commands can be found below.

---

Optional Background Information

It sounds like you're using BSD/macOS sed, whose -i option requires an option-argument that specifies the suffix of the backup file to create.
Therefore, it is your sed script that (against your expectations) is interpreted as -i's option-argument (the backup suffix), and your input filename is interpreted as the script, which obviously fails.

By contrast, your commands use GNU sed syntax, where -i can be used by itself to indicate that no backup file of the input file to updated in-place is to be kept.

The equivalent BSD sed option is -i '' - note the technical need to use a separate argument to specify the option-argument '', because it is the empty string (if you used -i'', the shell would simply strip the '' before sed ever sees it: -i'' is effectively the same as just -i).

Sadly, this then won't work with GNU sed, because it only recognizes the option-argument when directly attached to -i, and would interpret the separate '' as a separate argument, namely as the script.

This difference in behavior stems from a fundamentally differing design decision behind the implementation of the -i option and it probably won't go away for reasons of backward compatibility.^[1]

If you do not want a backup file created, there is no single -i syntax that works for both BSD and GNU sed.

There are four basic options:

• (a) If you know that you'll only be using either GNU or BSD sed, construct the -i option accordingly: -i for GNU sed, -i '' for BSD sed.

• (b) Specify a nonempty suffix as -i's option-argument, which, if you attach it directly to the -i option, works with both implementations; e.g., -i'.bak'. While this invariably creates a backup file with suffix .bak, you can just delete it afterward.

• (c) Determine at runtime which sed implementation you're dealing with and construct the -i option accordingly.

• (d) omit -i (which is not POSIX-compliant) altogether, and use a temporary file that replaces the original on success: sed '...' "$Deploy_path" > tmp.out && mv tmp.out "$Deploy_path".
  Note that this is in essence what -i does behind the scenes, which can have unexpected side effects, notably an input file that is a symlink getting replaced with a regular file; -i, does, however, preserve certain attributes of the original file: see the lower half of this answer of mine.

Here's a bash implementation of (c) that also streamlines the original code (single sed invocation with 2 substitutions) and makes it more robust (variables are double-quoted):

#!/bin/bash

RABBITMQ_HOST='rabbitmq1'
RABBITMQ_PASS='12345'
Deploy_path="test.env"

# Construct the Sed-implementation-specific -i option-argument.
# Caveat: The assumption is that if the `sed` is not GNU Sed, it is BSD Sed,
#         but there are Sed implementations that don't support -i at all,
#         because, as Steven Penny points out, -i is not part of POSIX.
suffixArg=()
sed --version 2>/dev/null | grep -q GNU || suffixArg=( '' )

sed -i "${suffixArg[@]}" '
 s/^\(RABBITMQ_HOST\)=.*/\1='"$RABBITMQ_HOST"'/
 s/^\(RABBITMQ_PASS\)=.*/\1='"$RABBITMQ_PASS"'/
' "$Deploy_path"

Note that with the specific values defined above for $RABBITMQ_HOST and $RABBITMQ_PASS, it is safe to splice them directly into the sed script, but if the values contained instances of &, /, \, or newlines, prior escaping would be required so as not to break the sed command.
See this answer of mine for how to perform generic pre-escaping, but you may also consider other tools at that point, such as awk and perl.

---

^{[1] GNU Sed considers the option-argument to -i optional, whereas BSD Sed considers it mandatory, which is also reflected in the syntax specs. in the respective man pages: GNU Sed: -i[SUFFIX] vs. BSD Sed -i extension.}

Answer 3

Simple Case

If test.env contains only the two variables, you can simply create a new file, or overwrite existing:

printf "RABBITMQ_HOST=%s\nRABBITMQ_PASS=%s\n" \
  "${RABBITMQ_HOST}" "${RABBITMQ_PASS}" > "$Deploy_path"

Fixing Unquoted Variables and Optimizing the SED Commands

Try to fix your command as follows:

sed -i -e 's/\(RABBITMQ_HOST=\).*/\1'"$RABBITMQ_HOST"'/' \
  -e 's/\(RABBITMQ_PASS=\).*/\1'"$RABBITMQ_PASS"'/' \
  "$Deploy_path"

You should enclose the variables in double quotes, since otherwise the shell will interpret the contents. In a content in double quotes, the shell will interpret only $ (replacing the variable with its content), backquote, and \ (escape). Also note the use of multiple -e options.

Why SED is Bad for this Task (in my Opinion)?

But, as it is said in @mklement0's answer, -i might not work in this form on BSD systems. Also, the command only modifies the two variables, if they are defined in $Deploy_path file, if the file exists. It will not add new variables into the file. Be warned, the variables are embedded directly into the replacement, and their values, generally, should be escaped according to the SED rules!

Alternative

If the test.env file is trusted, I recommend to load the variables, modify them and print to the output file:

(
  # Load variables from test.env
  source test.env

  # Override some variables
  RABBITMQ_HOST=rabbitmq1
  RABBITMQ_PASS=12345

  # Print all variables prefixed with "RABBITMQ_".
  # In POSIX mode, `set` will not output defines and functions
  set -o posix
  set | grep ^RABBITMQ_
) > "$Deploy_path"

Consider adjusting the file system permissions for test.env. I suppose, the source file is a trusted template.

The solution without SED is better, in my opinion, because the SED implementations may vary, and the in-place option may not work as expected on different platforms.

But, isn't source risky?

While parsing the shell variable assignments is usually an easy task, it is more risky than just sourcing the ready-for-use "script" (test.env). For instance, consider the following line in your test.env:

declare RABBITMQ_HOST=${MYVAR:=rabbitmq1}

or

export RABBITMQ_HOST=host

All of the currently suggested solutions, except the code using source, assume that you assign the variable as RABBITMQ_HOST=.... Some of the solutions even assume that RABBIT_HOST is placed at the beginning of the line. Ahh, you might fix the regular expression then, right? Just for this case...

Thus, source is risky as much as the file being sourced is not trusted. Think of #include <file> in C, or include "file.php" in PHP. These instructions include the source into the current source as well. So don't blindly consider sourcing a file as anti-pattern. It all depends on the particular circumstances. If your test.env is a part of your repository being deployed, then it is surely safe to call source test.env. That's my opinion, however.