Create and pipe a file-like object as input for a command

Question

I'm looking for a better way to do this, if possible:

import subprocess

f = open('temp.file', 'w+')
f.write('hello world')
f.close()

out = subprocess.check_output(['cat', 'temp.file'])

print out

subprocess.check_output(['rm', 'temp.file'])

In this example I'm creating a file and passing it as input to cat (in reality it's not cat I'm running but some other program that parses an input pcap file).

What I'm wondering is, is there a way in Python I can create a 'file-like object' with some content, and pipe this file-like object as input to a command-line program. If it is possible, I reckon it would be more efficient than writing a file to the disk and then deleting that file.

Answer 1

If the program is configured to read from stdin , you can use Popen.communicate :

>>> from subprocess import Popen, PIPE
>>> p = Popen('cat', stdout=PIPE, stdin=PIPE, stderr=PIPE)
>>> out, err = p.communicate(input=b"Hello world!")
>>> out
'Hello world!'

Answer 2

check_output takes a stdin input argument to specify a file-like object to connect to the process's standard input.

with open('temp.file') as input:
    out = subprocess.check_output(['cat'], stdin=input)

Also, there's no need to shell out to run rm ; you can remove the file directly from Python:

os.remove('temp.file')

Answer 3

You can write to a TemporaryFile

import subprocess
from tempfile import TemporaryFile
f = TemporaryFile("w")
f.write("foo")
f.seek(0)
out = subprocess.check_output(['cat'],stdin=f)

print(out)
b'foo'

If you just want to write to a file like object and get the content:

from io import StringIO
f = StringIO()
f.write("foo")

print(f.getvalue())

Answer 4

If the command accepts only filenames, if it doesn't read input from its stdin ie, if you can't use stdin=PIPE + .communicate() or stdin=real_file then you could try /dev/fd/# filenames:

#!/usr/bin/env python3
import os
import subprocess
import threading

def pump_input(pipe):
    with pipe:
        for i in range(3):
            print(i, file=pipe)

r, w = os.pipe()
try:
    threading.Thread(target=pump_input, args=[open(w, 'w')]).start()
    out = subprocess.check_output(['cat', '/dev/fd/'+str(r)], pass_fds=[r])
finally:
    os.close(r)
print('got:', out)

No content touches the disk. The input is passed to the subprocess via the pipe directly.

If you have a file-like object that is not a real file (otherwise, just pass its name as the command-line argument) then pump_input() could look like:

import shutil

def pump_input(pipe):
    with pipe:
        shutil.copyfileobj(file_like_object, pipe)