强制subprocess.Popen使用write（）函数而不是fileno（）将stdout / stderr写入python中的文件状对象

Question

My goal is to open a process using subprocess.Popen in python, and have this process pipe its stdout and stderr to a custom RingBuffer class that I've written, allowing me to periodically inspect the contents of the buffer from the same space I instantiated the subprocess from. This is important, I know there are ways to make a separate program, pipe the output of the subprocess to the stdin of that ringbuffer program, but then I have to go and manually inspect some underlying file that contains the ring buffer contents, etc, etc. The ideal thing would be to connect the output of the subprocess to some object that I have access to.

First, from the documenation for subprocess (python 2.X) ( https://docs.python.org/2/library/subprocess.html )

stdin, stdout and stderr specify the executed program's standard input, standard output and standard error file handles, respectively. Valid values are PIPE, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. With the default settings of None, no redirection will occur; the child's file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the child process should be captured into the same file handle as for stdout

"an existing file object", so I assume if I make a class that conforms to the file interface it should work, right?

Let's say I've made a class like this

class RingBuffer(object):

    def __init__(max_size=1024*1024):
      self.max_size = max_size
      self.current_size = 0


    def write(self, data):
        self.current_size += len(data)
        self.data.append(data)
        if self.current_size >= self.max_size_bytes:
            while self.current_size >= self.trim_size_bytes:
                try:
                    popped = self.data.pop()
                    self.current_size -= len(popped)
                except IndexError as e:
                    break

def writelines(self, sequence):
    for item in sequence:
        self.write(item)

def dumps(self):
    ret = [line for line in self.data]
    return '\n'.join(ret)

def clear(self):
    self.data.clear()
    self.current_size = 0

granted there are likely bugs in this program but you get the gist, it exposes a write() function and writes data to a circular buffer, trimming the buffer to a certain size when it gets too bug, and letting the user recover the data when they want with the dumps() function.

Now, if I try something like this

r = RingBuffer()
pr = subprocess.Popen(["timeout", "15", "yes"], stdout=r, stderr=subprocess.STDOUT)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 382, in __init__
    errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 818, in _get_handles
    c2pwrite = stdout.fileno()
AttributeError: 'RingBuffer' object has no attribute 'fileno'

okay, so my "file-like" object is missing the fileno() function to conform to the file interface. This is where the issue lays.. why does it need a fileno? Why can't it just use my supplied write() function? I'm assuming that it is going to by-pass my write function and instead use the fileno to write directly to the file?

Let's say I add in a stub of that function

def fileno()
    return None

then this happens

r = RingBuffer()
pr = subprocess.Popen(["timeout", "15", "yes"], stdout=r, stderr=subprocess.STDOUT)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

---

so my question is: how can I force subprocess.Popen to use my write() function for my file -like object instead of trying to write directly to the file handle returned from a non-existent fileno() function? If there is no way to do this.. is there any way to accomplish what I want here?

I know that theoretically I could make some file, /tmp/ringlog.txt , and open that file on instantiation of the class, then have the program write to that file, and have my program periodically look at the file and keep it under the max_size using a similar ringbuffer algorithm, but that's a mess.

Another option is to make a program that reads stdin, writes to a file, and ringbuffers the content to keep the file under a certain size, but then I'm still dealing with actual file, I just want to keep the contents in memory and accessible from the calling python environment.

Answer 1

The child process is going to write to its stdout using standard OS-level file writing calls, which means it needs something compatible with those calls. The child process can't see into Python's memory or call methods on Python objects.

If you want to write the subprocess's output to a file-like object that doesn't represent something the OS can treat as a file, you're going to have to receive the output through a pipe and write it to the file-like object yourself. You could spawn a worker thread for that (and make sure to synchronize access to the object, if you're planning to read from it before the worker terminates), but it might be simpler to interact with the pipe directly.

Answer 2

If you are going to work with subprocess.Popen then I suggest understanding the problems that come with using pipes, which is generally in the area of deadlocks.

See: https://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/