Why does subprocess.Popen.wait use a busy loop?

Question

As documented here , Popen.wait does busy wait. I believe some operating systems have system calls to wait for processes to end without the need for busy looping. Why aren't such mechanisms utilized?

Answer 1

from subprocess import Popen, PIPE

handle = Popen('ping -n 100 www.google.se', shell=True, stdout=PIPE)
while handle.poll() is None: # <-- Equivilant of .wait()
    print(handle.stdout.readline())

wait is a short-hand-function for .poll() which basically does the same thing except if you manually use .poll() in a holding loop, you can compute things during this process.

Normally it's used to dump out stdout/stderr stuff (which, if you don't might hold up the application or raise an exception).

Also using shell is risky, but it saves you a lot of headache when learning and testing.

The only way to truly not block anything at all (even the above approach "blocks" the code next in line) is to utelize threads:

from threading import *
from subprocess import Popen, PIPE

class worker(Thread):
    def __init__(self, command):
        Thread.__init__(self)
        self.handle = Popen(command, shell=True, stdout=PIPE)
        self.start()

    def run(self):
        while self.handle.poll() is None:
            print('Thread-output',self.handle.stdout.readline()) # Note, you might loose the last output.. just saying.

print('I\'m a panda..')
worker('ping -n 100 www.google.se')
print('And i work really well simultaneously...')

Useful tip when debugging:

from subprocess import Popen, PIPE, STDOUT
handle = Popen('ping -n 100 www.google.se', shell=True, stdout=PIPE, stderr=PIPE)
# You need to flush the output buffers:
handle.stderr.readline()
handle.stdout.readline()
# or
handle = Popen('ping -n 100 www.google.se', shell=True, stdout=PIPE, stderr=STDOUT)
handle.stdout.readline() # Does both stdout+stderr at the same time, saves you a line.

# and always close your open filehandles.
handle.stdout.close()
handle.stderr.close() # If you've separated both.

Regarding your operating system question

I think you might be referring to system services or daemons?
These types of "processes" as you describe them is specified as blocking or non-blocking (which are the terms you're looking for). The developer of each init-script for these processes determine if the process should be blocking until done (or timeout reached) or if the processes should be forked into the background.

Things that might be blocking would be OpenLDAP or the internal mail-transporter, while other processes such as OpenVPN or Apache might be forked into the background leaving the system enable to continue it's start-up sequence.

Answer 2

I'm not entirely positive. I don't think there's a way to have waitpid or equivalent have a completely-non-invasive, synchronous timeout. Also, I think some Unices have different rules for exactly how waitpid plays with signals.

The comments say they stole the loop from Thread.wait , and the comments in threading.py suggest it's used for responsiveness.