Subprocess and using the output file for another python script within Python script

Question

Using a single python script, I want the output of one program to be captured and consumed by a second program running in a separate process. I'm attempting to use subprocesses to run these two programs from within a single python script:

prog1 = 'python prog1.py'
args1 = 'input_1.txt'
proc1 = Popen(prog1 + ' ' + args1,stdout=PIPE,stderr=STDOUT, shell=True)
for i in proc1.stdout:
   print i
prog2 = 'python prog2.py'
args2 = i   # the input which is the output of prog1
proc2 = Popen(prog2 + ' ' + args2,stdout=PIPE,stderr=STDOUT, shell=True)
for m in proc2.stdout:
    print m

Answer 1

Sounds like you want shell pipeline https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline

p1 = Popen(["python", "prog1.py"], stdout=PIPE)
p2 = Popen(["python", "prog2.py"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()
output = p2.communicate()[0]

Answer 2

If prog2 accepts input only as a filename ie, if you need to emulate the bash process substitution: prog2 <(prog1 input_1.txt) then to avoid writing the output of prog1 subprocess to disk, you could use /dev/fd/N special filenames if your OS supports them:

#!/usr/bin/env python2
import sys
from subprocess import Popen, PIPE

prog1 = Popen([sys.executable, "prog1.py", "input_1.txt"], stdout=PIPE)
with prog1.stdout:
    cmd2 = [sys.executable, "prog2.py", "/dev/fd/%d" % prog1.stdout.fileno()]
    prog2 = Popen(cmd2, stdout=PIPE)
output = prog2.communicate()[0]
statuses = [prog1.wait(), prog2.returncode]

You should check whether prog2 supports a special filename '-' to read the input from stdin ie, to emulate the pipeline: prog1 input_1.txt | prog2 - prog1 input_1.txt | prog2 - :

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

prog2 = Popen([sys.executable, "prog2.py", "-"], stdin=PIPE, stdout=PIPE)
prog1 = Popen([sys.executable, "prog1.py", "input_1.txt"], stdout=prog2.stdin)
output = prog2.communicate()[0]
statuses = [prog1.wait(), prog2.returncode]

Or (better) import the modules instead of running them as subprocesses and run the corresponding functions, use multiprocessing if necessary. See Call python script with input with in a python script using subprocess