Using python subprocess to launch a python script in a different anaconda environment

Question

I need to launch a python script that requires a different environment from the python script that launches it. I'm currently attempting to use the subprocess module to create a new process and windows bat file that sets up the second Anaconda environment and then launches that script.

The rationale behind making this a two step process with different environments is that this tool needs to be run as a script tool in ESRI's ArcPro GUI. This software comes with an Anaconda installation and allows users to make their own environments and scripts that can be run from the GUI, however it forces these environments to have their own ArcPy package installed and I cannot find a combination of packages that allows my script to exist alongside ArcPy. Because of this I intend to use the default ArcPro Anaconda environment to run a script which records parameters to a text file, and then launches a subprocess which will use a bat file to set up a separate Anaconda environment and launch the second script.

My attempt at that final step currently looks like this:

subprocess.Popen("C:\\Windows\\System32\\cmd.exe C:\\Users\RDCHLNRO\\AppData\\Local\\Continuum\\anaconda3\\Scripts\\__TESTFORARC.bat")

This bat file for starting Anaconda was copied from Anaconda's own activate.bat:

@set "_args1=%1"
@set _args1_first=%_args1:~0,1%
@set _args1_last=%_args1:~-1%
@set _args1_first=%_args1_first:"=+%
@set _args1_last=%_args1_last:"=+%
@set _args1=
@if "%_args1_first%"=="+" if NOT "%_args1_last%"=="+" (
    @CALL "%~dp0..\condabin\conda.bat" activate
    @CALL conda activate myenv
    @CALL C:\Users\RDCHLNRO\AppData\Local\Continuum\anaconda3\envs\myenv\python.exe C:\Users\RDCHLNRO\Desktop\waves2021\cc_ARC.py
    
    @GOTO :End
)
@REM This may work if there are spaces in anything in %*
@CALL "%~dp0..\condabin\conda.bat" activate %*
@CALL conda activate myenv
@CALL C:\Users\RDCHLNRO\AppData\Local\Continuum\anaconda3\envs\myenv\python.exe C:\Users\RDCHLNRO\Desktop\waves2021\cc_ARC.py

:End
@set _args1_first=
@set _args1_last=

This results in a windows command prompt launching with the following result:

Traceback (most recent call last):
  File "C:\Users\RDCHLNRO\AppData\Local\Continuum\anaconda3\Scripts\conda-script.py", line 11, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

CommandNotFoundError: 'activate'

    Traceback (most recent call last):
      File "C:\Users\RDCHLNRO\Desktop\waves2021\cc_LZMST_ARC.py", line 5, in <module>
        import numpy as np
      File "C:\Users\RDCHLNRO\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\numpy\__init__.py", line 140, in <module>
        from . import core
      File "C:\Users\RDCHLNRO\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\numpy\core\__init__.py", line 101, in <module>
        from . import _internal
      File "C:\Users\RDCHLNRO\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\numpy\core\_internal.py", line 18, in <module>
        IS_PYPY = platform.python_implementation() == 'PyPy'
      File "C:\Users\RDCHLNRO\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\platform.py", line 1262, in python_implementation
        return _sys_version()[0]
      File "C:\Users\RDCHLNRO\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\platform.py", line 1223, in _sys_version
        repr(sys_version))
    ValueError: failed to parse CPython sys.version: '3.7.10 (default, Feb 26 2021, 13:06:18) [MSC v.1916 64 bit (AMD64)]'

However, if I start a windows command prompt through the windows start menu and point it towards my bat file, the Anaconda environment starts and the script is launched.

I have been unable to find any difference between the terminal that subprocess opened through the ArcPro GUI (the window persists after the error) and the terminal opened through the start menu aside from their ability to open this bat file. Both trace back to the same system32 folder.

I am sure there are more elegant ways of accomplishing this entire process, but at this point I would just like to figure out why these terminal windows, apparently coming from the same executable have such different behavior, and what I can do to get the desired behavior from the one launched by the subprocess within ArcPro.

Answer 1

The conda run command is a more idiomatic way of executing specific code within an environment without having to manage an interactive shell. Try something like:

subprocess.Popen(“conda run -n myenv python C:\Users\RDCHLNRO\Desktop\waves2021\cc_ARC.py“.split(“ “))

You may also need the --live-stream flag if the script has interactivity.