在 Python 中使用不同的可执行文件运行多处理池

Question

Is it possible to run a multiprocessing pool (or a subprocess) with a different executable than the main process in Python?

The use case I have in mind is running some one-off tests in parallel using different virtualenvs at the same time, for gathering some data for different library versions.

The issues I can forsee is that pickling/sharing things between subprocesses this way might not be trivial, as the library versions could be different. However, simple data types (lists, dicts) will probably work.

Answer 1

multiprocessing.spawn() supports changing the interpreter, but not concurrently.

If that's a problem, just a subprocess with stdin and stdout in binary mode and schlepping a pickle up and down the pipe (that sounded wrong) should do the trick... This assumes the parent is running Python 3.

import sys
import pickle
import subprocess
import random
import io

# Grab hold of binary I/O streams

if sys.version_info[0] == 3:
    i_stream = sys.stdin.buffer
    o_stream = sys.stdout.buffer
else:
    i_stream = sys.stdin
    o_stream = sys.stdout


def child():
    input_data = pickle.loads(i_stream.read())
    output_data = dict(
        input_data,
        random_twice=input_data["random"] * 2,
        version=tuple(sys.version_info),
    )
    o_stream.write(pickle.dumps(output_data, protocol=2))


def parent():
    # The highest protocol supported by Python 2.7 is version 2.
    input_data = pickle.dumps(
        {"hello": "world", "random": random.randint(1, 100000)},
        protocol=2,
    )
    for python in ("python2.7", "python3.7"):
        proc = subprocess.run(
            ["/usr/bin/env", python, sys.argv[0], "child"],
            check=True,
            encoding=None,
            input=input_data,
            stdout=subprocess.PIPE,
        )
        print(python, pickle.loads(proc.stdout))


if __name__ == "__main__":
    if "child" in sys.argv:
        child()
    else:
        parent()

outputs

python2.7 {'random_twice': 174850, 'random': 87425, 'hello': 'world', 'version': (2, 7, 16, 'final', 0)}
python3.7 {'hello': 'world', 'random': 87425, 'random_twice': 174850, 'version': (3, 7, 6, 'final', 0)}