Error calling subprocess on python when redirecting stdout

Question

I tried to filter a file that was generated by a function in a python script:

out = subprocess.check_output(["sed","-n","'s/pattern/&/p'",oldFile,">",newFile])

However, I got the followong error about my command:

returned non-zero exit status 1

What is wrong?

Answer 1

As devnull stated, the > is interpreted by the shell. Since it is better to avoid using shell=True , use the stdout parameter instead:

import subprocess
with open(newFile, 'w') as newFile:
    subprocess.check_call(
        ["sed", "-n", "s/S/&/p", oldFile], stdout=newFile)

Answer 2

You are using > redirections, which require a shell to interpret the syntax.

When you are redirecting the output of sed , there is no point is using check_output here. Use subprocess.call() or subprocess.check_call() instead and verify the return code.

Either run the command through the shell:

import pipes

out = subprocess.call("sed -n 's/S/&/p' {} > {}".format(
    pipes.quote(oldFile), pipes.quote(newFile), shell=True)

or use a pipe:

with open(newFile, 'w') as pipetarget:
    out = subprocess.call(["sed", "-n", "s/S/&/p", oldFile],
                                  stdout=pipetarget)

Note that you shouldn't use quotes on the 's/S/&/p' string when used as a separate argument in the argument list; when not passing that to the shell it doesn't need escaping from shell parsing either.