How do I run a sub-process, display its output in a GUI and allow it to be terminated?

Question

I have been trying to write an application that runs subprocesses and (among other things) displays their output in a GUI and allows the user to click a button to cancel them. I start the processes like this:

queue = Queue.Queue(500)
process = subprocess.Popen(
    command,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT)
iothread = threading.Thread(
    target=simple_io_thread,
    args=(process.stdout, queue))
iothread.daemon=True
iothread.start()

where simple_io_thread is defined as follows:

def simple_io_thread(pipe, queue):
    while True:
        line = pipe.readline()
        queue.put(line, block=True)
        if line=="":
            break

This works well enough. In my UI I periodically do non-blocking "get"s from the queue. However, my problems come when I want to terminate the subprocess. (The subprocess is an arbitrary process, not something I wrote myself.) I can use the terminate method to terminate the process, but I do not know how to guarantee that my I/O thread will terminate. It will normally be doing blocking I/O on the pipe. This may or may not end some time after I terminate the process. (If the subprocess has spawned another subprocess, I can kill the first subprocess, but the second one will still keep the pipe open. I'm not even sure how to get such grand-children to terminate cleanly.) After that the I/O thread will try to enqueue the output, but I don't want to commit to reading from the queue indefinitely.

Ideally I would like some way to request termination of the subprocess, block for a short (<0.5s) amount of time and after that be guaranteed that the I/O thread has exited (or will exit in a timely fashion without interfering with anything else) and that I can stop reading from the queue.

It's not critical to me that a solution uses an I/O thread. If there's another way to do this that works on Windows and Linux with Python 2.6 and a Tkinter GUI that would be fine.

---

EDIT - Will's answer and other things I've seen on the web about doing this in other languages suggest that the operating system expects you just to close the file handle on the main thread and then the I/O thread should come out of its blocking read. However, as I described in the comment, that doesn't seem to work for me. If I do this on the main thread:

process.stdout.close()

I get:

IOError: close() called during concurrent operation on the same file object.

...on the main thread. If I do this on the main thread:

os.close(process.stdout.fileno())

I get:

close failed in file object destructor: IOError: [Errno 9] Bad file descriptor

...later on in the main thread when it tries to close the file handle itself.

Answer 1

This is probably old enough, but still usefull to someone coming from search engine...

The reason that it shows that message is that after the subprocess has been completed it closes the file descriptors, therefore, the daemon thread (which is running concurrently) will try to use those closed descriptors raising the error.

By joining the thread before the subprocess wait() or communicate() methods should be more than enough to suppress the error.

my_thread.join()
print my_thread.is_alive()
my_popen.communicate()

Answer 2

I know this is an old post, but in case it still helps anyone, I think your problem could be solved by passing the subprocess.Popen instance to io_thread, rather than it's output stream. If you do that, then you can replace your while True: line with while process.poll() == None: .

process.poll() checks for the subprocess return code; if the process hasn't finished, then there isn't one (ie process.poll() == None ). You can then do away with if line == "": break .

The reason I'm here is because I wrote a very similar script to this today, and I got those:-
IOError: close() called during concurrent operation on the same file object. errors.

Again, in case it helps, I think my problems stem from (my) io_thread doing some overly efficient garbage collection, and closes a file handle I give it (I'm probably wrong, but it works now..) Mine's different tho in that it's not daemonic, and it iterates through subprocess.stdout, rather than using a while loop.. ie:-

def io_thread(subprocess,logfile,lock):
    for line in subprocess.stdout:
        lock.acquire()
        print line,
        lock.release()
        logfile.write( line )

I should also probably mention that I pass the bufsize argument to subprocess.Popen, so that it's line buffered.

Answer 3

在终止进程的代码中，您还可以显式地os.close()您的线程正在读取的管道？

Answer 4

You should close the write pipe instead... but as you wrote the code you cannot access to it. To do it you should

1. crate a pipe
2. pass the write pipe file id to Popen 's stdout
3. use the read pipe file simple_io_thread to read lines.

Now you can close the write pipe and the read thread will close gracefully.

queue = Queue.Queue(500)
r, w = os.pipe()
process = subprocess.Popen(
    command,
    stdout=w,
    stderr=subprocess.STDOUT)
iothread = threading.Thread(
    target=simple_io_thread,
    args=(os.fdopen(r), queue))
iothread.daemon=True
iothread.start()

Now by

os.close(w)

You can close the pipe and iothread will shutdown without any exception.