Actual meaning of 'shell=True' in subprocess

Question

I am calling different processes with the subprocess module. However, I have a question.

In the following codes:

callProcess = subproces         


        

Answer 1

The other answers here adequately explain the security caveats which are also mentioned in the subprocess documentation. But in addition to that, the overhead of starting a shell to start the program you want to run is often unnecessary and definitely silly for situations where you don't actually use any of the shell's functionality. Moreover, the additional hidden complexity should scare you, especially if you are not very familiar with the shell or the services it provides.

Where the interactions with the shell are nontrivial, you now require the reader and maintainer of the Python script (which may or may not be your future self) to understand both Python and shell script. Remember the Python motto "explicit is better than implicit"; even when the Python code is going to be somewhat more complex than the equivalent (and often very terse) shell script, you might be better off removing the shell and replacing the functionality with native Python constructs. Minimizing the work done in an external process and keeping control within your own code as far as possible is often a good idea simply because it improves visibility and reduces the risks of -- wanted or unwanted -- side effects.

Wildcard expansion, variable interpolation, and redirection are all simple to replace with native Python constructs. A complex shell pipeline where parts or all cannot be reasonably rewritten in Python would be the one situation where perhaps you could consider using the shell. You should still make sure you understand the performance and security implications.

In the trivial case, to avoid shell=True, simply replace

subprocess.Popen("command -with -options 'like this' and\\ an\\ argument", shell=True)

with

subprocess.Popen(['command', '-with','-options', 'like this', 'and an argument'])

Notice how the first argument is a list of strings to pass to execvp(), and how quoting strings and backslash-escaping shell metacharacters is generally not necessary (or useful, or correct). Maybe see also When to wrap quotes around a shell variable?

If you don't want to figure this out yourself, the shlex.split() function can do this for you. It's part of the Python standard library, but of course, if your shell command string is static, you can just run it once, during development, and paste the result into your script.

As an aside, you very often want to avoid Popen if one of the simpler wrappers in the subprocess package does what you want. If you have a recent enough Python, you should probably use subprocess.run.

• With check=True it will fail if the command you ran failed.
• With stdout=subprocess.PIPE it will capture the command's output.
• With text=True (or somewhat obscurely, with the synonym universal_newlines=True) it will decode output into a proper Unicode string (it's just bytes in the system encoding otherwise, on Python 3).

If not, for many tasks, you want check_output to obtain the output from a command, whilst checking that it succeeded, or check_call if there is no output to collect.

I'll close with a quote from David Korn: "It's easier to write a portable shell than a portable shell script." Even subprocess.run('echo "$HOME"', shell=True) is not portable to Windows.