How do I tell POpen to use/set certain environment variables?

Question

I'm using Python 3.7 and Django. I use the below to run a command I would normally run in the shell ...

out = Popen([settings.SELENIUM_RUNNER_CMD, file_path], stderr=STDOUT, stdout=PIPE)
t = out.communicate()[0], out.returncode

his dies with the error

b'env: node: No such file or directory\n'

What I'm trying to figure out is how to give my Python environment access to the normal environment variables I have access to, or figure out a way to set them before I run my Python command. Normally "node" is easily found when I check as myself

davea$ which node
/usr/local/bin/node

But I don't know how to tell Python to use the same PATH that I have access to.

Answer 1

If we refer to Popen's documentation , we can see three relevant arguments:

1. cwd str or path -like object, that's the current working directory
2. env mapping (let's say a dict ), that's the environment mapping passed to the called program
3. shell flag, whether you wrap the program inside of a shell or not

Let's review each solution.

---

If you can afford it, just use cwd="where is node" , for instance, if node is in /usr/local/bin , you can just use cwd=/usr/local/bin or cwd=os.path.join(USR_LOCAL, 'bin') for example. But, everything will be created in this folder, which might not be what you wish for (logs, assumptions on the current working directory).

---

Now, for the environment:

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of the default behavior of inheriting the current process' environment. It is passed directly to Popen.

You can just copy your current environment through os.environ and add something in the PATH like this:

new_env = os.environ.copy()
new_env['PATH'] = '{}:/usr/local/bin'.format(new_env['PATH'])

Then pass this new_env mapping and there you are!

---

If you really want to rely on shell, you can, but here's the platform-details:

POSIX-platforms

On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of: Popen(['/bin/sh', '-c', args[0], args[1], ...])

Windows platforms

On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (eg dir or copy). You do not need shell=True to run a batch file or console-based executable.

You can use something like PATH=whatever and use your whole shell-fu directly, but caveats are: security considerations .

---

Bonus solution

Just re-define PATH before calling your Python process. If you're using Django, you're either using:

1. The development server
2. A production-grade server

In both cases, all you have to do is to re-define the environment of the parent process, for a production-grade server such as Gunicorn, this is possible and there is documentation to do it. For a development server, do it at your own shell level (but warning! You might have to document such a behavior or tell anyone using your software you're assuming node is in the path which is… most of the time fair).

Answer 2

os.environ.copy() is best for what you're looking for.

import subprocess, os
my_env = os.environ.copy()
out = Popen([settings.SELENIUM_RUNNER_CMD, file_path], stderr=STDOUT, stdout=PIPE, env=my_env)
t = out.communicate()[0], out.returncode

And that should be it!