Process substitution not allowed by Python's subprocess with shell=True?

Question

Here is a toy example of process substitution that works fine in Bash:

$ wc -l <(pwd)
1 /proc/self/fd/11

So why does the same command give a syntax error when invoked from Python's subprocess with shell=True?

>>> subprocess.check_call('wc -l <(pwd)', shell=True)
/bin/sh: 1: Syntax error: "(" unexpected
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/my/python/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'wc -l <(pwd)' returned non-zero exit status 2

Answer 1

/bin/sh: 1: Syntax error: "(" unexpected

You have a bashism . It is not valid according to POSIX, which is what /bin/sh implements.

Answer 2

An alternate solution is to shift more of the shell code to Python itself. For example:

from subprocess import Popen, PIPE, check_call

p1 = Popen(["pwd"], stdout=PIPE)
p2 = check_call(["wc", "-l"], stdin=p1.stdout)

This could often be the first step towards eliminating the need to use subprocess at all, as it decomposes the work into smaller chunks for which you may more readily see how to do in Python itself.

Answer 3

If you want to use Bash features (arrays, command substitution, here strings, or a lot of other non-POSIX extensions and enhancements ), you need to explicitly override the default shell:

subprocess.check_call(
    'wc -l <(pwd)',
    executable='/bin/bash',  # the beef
    shell=True)

or - somewhat more clumsily - run an explicit Bash instance:

subprocess.check_call(
    ['/bin/bash', '-c', 'wc -l <(pwd)'])

Notice how in the latter case we avoid separately specifying shell=True , and pass in the script as a list of strings (where the third string is an arbitrarily complex and/or long script as the argument to bash -c ).

(Actually there is a length limit. If your command line is longer than the kernel constant ARG_MAX you'll need to pass the script in a file or as standard input to the shell instead. On any modern system, we are talking megabytes of script, though.)

Running complex shell scripts (Bash or otherwise) from Python is dubious, anyway; you'll want to delegate absolutely as little as possible to a subprocess and take it from there in native Python code.