How to avoid loading wrong libraries when using a subprocess.Popen() from a python script to run a venv?

Question

I want to run a script using a venv python~3.9 from a subprocess call of another application that uses python3.6. However the imported libraries are wrong and from the site-packages of 3.6 version. How can I modify the subprocess call to load the correct libraries ie from the venv(3.9 version)

p = Popen([process_name, parameters_l], stdin=PIPE, stdout=PIPE, stderr=PIPE)

I have tried using the cwd and also changing the working directory via os.chdir however that doesn't seem to work. Furthermore I tried to run activat.bat from the venv, but the issue persists.

Answer 1

You should use an absolute path to that venv like /path/to/venv/bin/python3.9

Answer 2

This is my example.

example.py

this code show python version you run.

import sys 
print("version {}".format(sys.version))

traverse_python.sh

this code traverses various python versions to show what version runs the code.

you can change the process_name to the python version you want to use.

import time
import subprocess, os

pythoninterpretor = ['/usr/local/bin/python3.10', '/usr/local/bin/python3.11']

for i in pythoninterpretor:
    process_name = i
    parameters = 'example.py'
    p = subprocess.Popen([process_name, parameters])
    time.sleep(10)

my results

FYI, the result won't change even if you explicitly use another python version in shebang in the script.

version 3.10.8 (main, Oct 13 2022, 10:17:43) [Clang 14.0.0 (clang-1400.0.29.102)]
version 3.11.0 (main, Oct 25 2022, 14:13:24) [Clang 14.0.0 (clang-1400.0.29.202)]

In your case,

python_interpretor = '/absolute/path/env/python3.9'
p = Popen([python_interpretor, process_name, parameters_l], stdin=PIPE, stdout=PIPE, stderr=PIPE)

and you need to replace /absolute/path/env/python3.9 with your ideal version's path.

Answer 3

In order to solve the issue I had to get the current environment and delete some variables eg:

env = os.environ.copy()
del env["PYTHONPATH"]

Popen([process_name, parameters_l], env=env)

Finally use the new env in the subprocess call as a parameter. Hope it helps if anyone is facing a similar situation.