Python subprocess call hangs

Question

Python version: 2.6.7 I have the following subprocess.call within a for loop which is exectuted 18 times, however, the process constantly hangs on the 19th loop:

if config.get_bool_option(NAME, 'exclude_generated_code', True):
            for conf in desc.iter_configs():
                for gen in desc.iter_generators(conf):
                    generator.initialize_generated_path(gen, desc)
                    for genpath in gen.generated_path:
                        os.rename(cov_file, cov_file+'.temp')
                        exclude = ['lcov']
                        exclude += ['-r', cov_file+'.temp', '"'+genpath+'/*"']
                        exclude += ['-o', cov_file]
                        if verbose: Tracer.log.info("Running "+ ' '.join(exclude))
                        try:
                            subprocess.call(' '.join(exclude), stdout=out, stderr=out, shell=True)
                        except subprocess.CalledProcessError, e:
                        if verbose: Tracer.log.info("TESTING: Got Exception \n") 

The console output looks as follows:

Running lcov -r /remote/XXXXXX/coverage.19.temp "/remote/XXXXXX/xml/2009a/generated/*" -o /remote/XXXXX/gcov/coverage.19

Since I am not very familiar with python scripts, I just wandered whether I am doing something wrong here...I suspect a deadlock somewhere..

Would the stdout, stderr = process.communicate() deal with these issues?

Any expert answer on in which cases would the subprocess.call hang please? Thanks very much

Answer 1

When using subprocess, I tend to do something like this:

is_running = lambda: my_process.poll() is None

my_process = subprocess.Popen(' '.join(exclude), 
                              stdout=subprocess.PIPE, 
                              stderr=subprocess.PIPE,
                              shell=True)

# Grab all the output from stdout and stderr and log it
while is_running():
    rlist, wlist, xlist = select.select([my_process.stdout, my_process.stderr], [], [], 1)

# Log stdout, but don't spam the log
if my_process.stdout in rlist and verbose:
    # Adjust the number of bytes read however you like, 1024 seems to work 
    # pretty well for me. 
    Tracer.log.debug(my_process.stdout.read(1024))

# Log stderr, always
if my_process.stderr in rlist:
    # Same as with stdout, adjust the bytes read as needed.
    Tracer.log.error(my_process.stderr.read(1024))

I've seen the stdout stuff just dump a bunch of empty lines in my logs in the past, which is why I log that at the debug level. That prints to my logs during the development, but never gets written in production, so I can safely leave it in the code for debugging without putting garbage in their logs.

Hopefully, this can help expose just where your program is hanging and what's causing it.