How to choose the python interpreter of a ProcessPoolExecutor process?

Question

I am making a program that runs processes in a ProcessPoolExecutor and returns the result once finished. The script I want to run uses pretty old libraries, so I don't want to have to include them in the main script. Instead, I have another virtual environment that is set up to run the subprocess.

I am using a ProcessPoolExecutor to spawn the jobs. How can I pick the python interpreter to use when running these jobs?

I saw that ProcessPoolExecutor has a initargs argument, but when I included it in my code:

with concurrent.futures.ProcessPoolExecutor(
        initargs=('PYTHONHOME', r'C:\Users\Tom.Mclean\Anaconda3\envs\weatherrouting_v1')) as pool:
    return await loop.run_in_executor(pool, fn, *args)

It just crashed.

EDIT:

with concurrent.futures.ProcessPoolExecutor() as pool:
    pool._mp_context.set_executable(r'C:\Users\Tom.Mclean\Anaconda3\envs\weatherrouting_v2\python.exe')
    return await loop.run_in_executor(pool, fn, *args)

Answer 1

Provide a multiprocessing context with explicit executable to the Pool.

import multiprocessing
import concurrent.futures

if __name__ == "__main__":
    context = multiprocessing.get_context('spawn')
    context.set_executable(...)  # <<< set worker executable

    with concurrent.futures.ProcessPoolExecutor(mp_context=context) as pool:
        ...

Note that concurrent.futures initialises the worker processes based on the parent process' concurrent.futures library. This means the executable must be able to parse and run the library in the parent process ' version.
As a result, this can eg be used with a different venv using the same Python version. It cannot be used to run a significantly older version of Python, such as a parent process of Python 3.9 and worker process of Python 3.7.