Python subprocess.Popen: redirection with ">" does not work

Question

The following code does not work properly in Python. The problem is that the output does not get redirected to output by using > :

command = toolpath + " " + query + " " + number + " > " + output;
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE);
output = process.communicate()[0];

The command is actually the following if you print it:

./subsetA.pl ./remainingqueries.fasta 100 > ./tmpQuery.fasta

So the perl script subsetA.pl takes two arguments and writes it to stdout which gets redirected to tmpQuery.fasta . But tmpQuery.fasta is empty after the command gets called.

If I run it direct on CLI then it works perfectly.

Answer 1

You don't need the shell's output redirection operator with Popen ; that's what the stdout argument is for.

command = [toolpath, query, number]
with open(output) as output_fh:
    process = subprocess.Popen(command, stdout=output_fh)

Since you are calling communicate to get the standard output, though, you don't want to redirect the output at all:

command = [toolpath, query, number]
process = subprocess.Popen(command, stdout=subprocess.PIPE)
output = process.communicate()[0]

Answer 2

You can try

command = toolpath + " " + query + " " + number + " > " + output;
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE,Shell=True);
output = process.communicate()[0];

Answer 3

Both current answers don't work!

This works (tested on Python 3.7):

subprocess.Popen(['./my_script.sh arg1 arg2 > "output.txt"'],
                 stdout=subprocess.PIPE, shell=True)

Note:

1. You do not need split or array in Popen.
2. BOTH stdout and shell parameters are needed.