Differences in sys.path when python2.7 is invoked normally or via subprocess

Question

Question

Why is it that python2.7 when called using a subprocess via python3 does not have the same sys.path as python2.7 called normally? Specifically, python2.7 via subprocess does not have the "/path/to/site-packages/" directory in sys.path.

Context

I'd like to use fabric to deploy a Django app I'm writing. My problem is that I've written the app in python3 , but fabric doesn't have explicit python3 support yet. My workaround, until fabric is fully compatible with python3 , is to call the fab script using subprocess .

For some reason when I call python2.7 using subprocess via python3 , I don't have access to any modules in site-packages .

python2.7 checks

I've got python2.7 and fabric==1.10.0 installed via Enthought.

$ which python
/Users/.../Library/Enthought/Canopy_32bit/User/bin/python

$ python --version
Python 2.7.6 --  32-bit

$ which fab
/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab

$ fab --version
Fabric 1.10.0
Paramiko 1.15.1

subprocess checks

I have no problem calling fab from within python2.7 using subprocess.

$ python
Enthought Canopy Python 2.7.6 | 32-bit | (default, Apr 11 2014, 12:06:39)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('fab --version', shell=True)
'Fabric 1.10.0\nParamiko 1.15.1\n'

I also have no problem calling python2.7 from within python3 using subprocess.

$ python3
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output('which python', shell=True)
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/python\n'
>>> subprocess.check_output('python --version', shell=True)
Python 2.7.6 --  32-bit
b''

DistributionNotFound: Fabric==1.10.0

However, even though my subprocess of python2.7 can "find" the fab script, I can't call it.

# python3
>>> subprocess.check_output(['which', 'fab'])
b'/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab\n'
>>> subprocess.check_output(['fab', '--version'])
Traceback (most recent call last):
  File "/Users/.../Library/Enthought/Canopy_32bit/User/bin/fab", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 2877, in <module>
    working_set.require(__requires__)
  File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 698, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Applications/Canopy.app/appdata/canopy-1.4.0.1938.macosx-x86/Canopy.app/Contents/lib/python2.7/site-packages/pkg_resources.py", line 596, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Fabric==1.10.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 620, in check_output
    raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['fab', '--version']' returned non-zero exit status 1

site-packages not in sys.path

It appears that python2.7 when called using subprocess via python3 does not have the same sys.path as python2.7 called normally.

As expected, sys.path did not have the Enthought "site-packages" directory, which contains the fabric module.

# python3
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## does not contain '/path/to/Enthought/python2.7/site-packages'

Manually add site-packages to sys.path

To confirm that it's possible: when I manually add the correct "site-packages" directory, I can successfully import fabric .

# python3
>>> subprocess.check_output('python -c\
    "import sys; sys.path.append(\'/path/to/Enthought/site-packages\');\
    from fabric import version; print version.get_version()"',\
    shell = True)
b'1.10.0\n'

Other options?

There's got to be a better way to make sure that python2.7, when invoked via subprocess from python3, has the same sys.path as python2.7 invoked normally. Can someone more familiar with subprocess weigh in?

Additional thoughts

It's really interesting that python2.7 can spawn another python2.7 via subprocess and that subprocess has the correct site-packages dir in sys.path.

$ python
>>> import subprocess
>>> subprocess.check_output('python -c "import sys; print sys.path"', shell=True)
## contains "/path/to/Enthought/python2.7/site-packages"

I also compared the sys.path 's from python3, python3 subprocessed by python3, and python3 subprocessed by python2.7, and was a bit surprised to find that all three resulted in the same sys.path .

Answer 1

subprocess supports an env parameter that, if given, will be the environment for the called command -- so make a copy of it, remove any troublesome variables, and pass that copy to subprocess :

my_env = os.environ.copy()
del my_env['__PYENV_LAUNCHER__']
subprocess.check_output(..., env=my_env)