Prevent creating new child process using subprocess in Python

Question

I need to run a lot of bash commands from Python. For the moment I'm doing this with
subprocess.Popen(cmd, shell=True)

Is there any solution to run all these commands in the same shell? subprocess.Popen opens a new shell at every execution and I need to set up all the necessary variables at every call, in order for cmd command to work properly.

Answer 1

subprocess.Popen lets you supply a dictionary of environment variables, which will become the environment for the process being run. If the only reason you need shell=True is to set an environment variable, then I suggest you use an explicit environment dictionary instead; it's safer and not particularly difficult. Also, it's usually much easier to construct command invocations when you don't have to worry about quoting and shell metacharacters.

It may not even be necessary to construct the environment dictionary, if you don't mind having the environment variables set in the running process. (Most of the time, this won't be a problem, but sometimes it is. I don't know enough about your application to tell.)

If you can modify your own environment with the settings, just do that:

os.environ['theEnvVar'] = '/the/value' 

Then you can just use a simple Popen.call (or similar) to run the command:

output = subprocess.check_output(["ls", "-lR", "/tmp"])

If for whatever reason you cannot change your own environment, you need to make a copy of the current environment, modify it as desired, and pass it to each subprocess.call :

env = os.environ.copy()
env['theEnvVar'] = '/the/value'
output = subprocess.check_output(["ls", "-lR", "/tmp"], env=env)

If you don't want to have to specify env=env every time, just write a little wrapper class.

Answer 2

Why not just create a shell script with all the commands you need to run, then just use a single subprocess.Popen() call to run it? If the contents of the commands you need to run depend on results calculated in your Python script, you can just create the shell script dynamically, then run it.

Answer 3

Use multiprocessing instead, it's more lightweight and efficient.

Unlike subprocess.Popen it does not open a new shell at every execution.

You didn't say you need to run subprocess.Popen and you may well not need to; you just said that's what you're currently doing. More justification please.

See set env var in Python multiprocessing.Process for how to set your env vars once and for all in the parent process.