使用 subprocess.call 运行 bash 脚本时，如何访问 anaconda3/bin 目录中的函数？

Question

I have the following problem: I wrote a bash script for data analysis that works perfectly fine when I run it from the terminal. To further automate the process I wanted to use a python script that runs the bash script (using subprocess.call), changes the working directory, and reruns the script (and so on). This also worked fine when I did it on my MacBook. However, I need to do the analysis on a Linux machine and here the problem occurred. Again, running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis. The functions are stored inside the anaconda3/bin folder. (Python does not even find other functions like "pip")

Of course, I could add the path to all the functions in the bash script but this seems very inefficient to me. So my question is: is there any better way of telling python where to look for the functions? And can you maybe explain to me why running the script from the terminal works but not when I use subprocess.call?

Here is the python script:

import subprocess
import os

path_list = ["Path1",
             "Path2"
             ]

for path in path_list:
    os.chdir(path)
    subprocess.call("Users/.../bash_script", shell=True)

Answer 1

I'm just posting my series of comments as an answer since I think this at least constitutes a reasonable answer for anyone running into a similar issue (your question could definitely be common enough to index from search engine results).

Issue:

...running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis

In general, you can troubleshoot this kind of problem with:

import subprocess
subprocess.call('echo $PATH', shell=True)

If the directory that contains the relevant binaries/scripts/etc. is not in the output, then you are facing a PATH issue in the shell created by subprocess.call .

The exact problem as confirmed by the OP in comments is that anaconda3/bin is not part of your PATH . Your script works in a regular terminal session because of the Anaconda initialization function that gets added to your .bashrc when installing.

Part of an answer that is very helpful here: Python - Activate conda env through shell script

The problem with your script, though, lies in the fact that the.bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.

Solution 1: Manually use the Anaconda sourcing function in your script

As the OP mentioned in the comments, their workaround was to use the initialization function added to their .bashrc in the script they are trying to run. Although this perhaps feels like not a great solution, this is a "good enough" workaround. Unfortunately I don't use Anaconda on Linux so I don't have an exact snippet of what this looks like. See the next section for a possibly "cleaner" solution.

Solution 2: Use bash -i to run your script

As mentioned in the same answer linked above, you might be able to use:

bash -i Users/.../bash_script

This will tell bash to run in interactive mode, which then properly sources your .bashrc file when creating the shell. As a result, Anaconda and related functions should work properly.

Solution 3: Manually add anaconda3/bin to PATH

You can check out this answer to decide if this is something you want to do. Keep in mind they are speaking about a Windows OS but most of the same applies to Linux.

When you add the directory to your PATH , you are specifically telling your system to always look in that directory for commands when executing by name, eg ping or which . This can have unexpected behavior if you have conflicts (eg a command is found with the same name in /usr/bin and .../anaconda3/bin ), and as such Anaconda does not add its bin folder to your PATH by default.

This is not necessarily "dangerous" per se, it's just not an ideal solution for most people. However, you are the boss of your own system. If you decide this works for your particular workflow, you can just add the export to your script:

export PATH="path/to/anaconda3/bin:$PATH"

This will set the PATH for use in the current shell and sub-processes.

Solution 4: Manually source the conda script (possibly outdated)

As mentioned in this answer , you can also opt to manually source the conda.sh script (keep in mind your conda.sh might be in another directory):

source /opt/anaconda/etc/profile.d/conda.sh

This will essentially run that shell script and add the included functionality to the current shell (eg the one spawned by subprocess.call ).

Keep in mind this answer is quite a bit older (~2013) and may not apply anymore, depending how much conda has changed over the years.

---

Notes

As I mentioned in the comments, you may want to post some related questions on https://unix.stackexchange.com/ . You have an interesting configuration challenge that may be better suited for answers specifically pertaining to Linux, since your issue is sourcing directly from Linux shell behavior.