确保subprocess.Popen调用的执行顺序

Question

I'm using subprocess.Popen to create an automated build script for the Scipy stack.

My current process follows below.

mathbuild.json:

{"suitesparse": {"version": "4.2.1",
         "dependencies": ["metis"],
         "downloads": ["http://www.cise.ufl.edu/research/sparse/SuiteSparse/SuiteSparse-4.2.1.tar.gz"],
         "build": ["cd $DL_DIR",
               "tar xvfz SuiteSparse-4.2.1.tar.gz",
               "cd SuiteSparse",
               "cp -r $DL_DIR/metis-4.0.3 metis-4.0.3"]},

 "metis": {"version": "4.0.3",
       "dependencies": [],
       "downloads": ["http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/OLD/metis-4.0.3.tar.gz"],
       "build": ["cd $DL_DIR",
             "tar xvfz metis-4.0.3.tar.gz",
             "cd metis-4.0.3",
             "make"]}}

mathbuild.py:

def package_list(package, config):
    for dependency in config[package]['dependencies']:
        yield from package_list(dependency, config)
    yield package

def build_package(package, config):
    command = '; '.join(config[package]['build'])
    build = subprocess.Popen(command, shell=True)


def process_package(package, config, env_dir, dl_dir):
    print('INSTALLING {0}'.format(package))
    print('Downloading...')
    download_package(package, config, dl_dir)
    print('Building...')
    build_package(package, config)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Install Pylab in a new venv.')
    parser.add_argument('env_dir', help='target directory for new environment')
    args = parser.parse_args()
    os.environ['ENV_DIR'], os.environ['DL_DIR'] = create_venv(args.env_dir)
    with open('mathbuild.json') as f:
        cfg = json.load(f)
    processed = []
    for package in package_list('suitesparse', cfg):
        if package not in processed:
            process_package(package, cfg,
                            os.environ['ENV_DIR'],
                            os.environ['DL_DIR'])
            processed += [package]

It creates a list of dependencies (such that later items depend on earlier ones) and then processes each one (downloading and then building based on the commands in the json file).

The issue is that packages are being built via new subprocess.Popen calls before dependencies are fully built. In the above example, the suitesparse execution begins even before the metis build is done. I assume that's because I'm opening a new subprocess each time around the for package in package_list('suitesparse', cfg) loop without regard for whether or not the prior subprocesses are done.

The question What's the best way to synchronize the loop-based Popen calls so that each call starts only when the previous call to Popen (ie previous item in the list) is done?

What I've tried I've tried changing the loop so that it builds one combined Popen (with both package builds), but that seems hackish.

Answer 1

It looks like you want subprocess.check_call() instead of Popen. From the docs :

Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError.

Your build function would look something like:

def build_package(package, config):
    command = '; '.join(config[package]['build'])
    subprocess.check_call(command, shell=True)

If you're actually using the Popen object, you can call the wait() method to wait for the subcommand to finish:

def build_package(package, config):
    command = '; '.join(config[package]['build'])
    build = subprocess.Popen(command, shell=True)
    # do something with the build object
    build.wait()
    # command is done