ModuleNotFoundError error when running a python file from shell script

Question

I have a python file download.py in which I am importing a module opc

Now, this python file invoked from a shell script build.sh like:

python3 $(pwd)/scripts/download.py -d $dir_name

When I am running the shell script, I am getting the following error:

Traceback (most recent call last):
  File "/home/ray/Desktop/repo/image-builder/scripts/download.py", line 14, in <module>
    from opc.utils import file_utils, logger
ModuleNotFoundError: No module named 'opc'

This is how shell script is invoked:

sudo sh scripts/src/build.sh

I have a virtual environment active where I have installed this package. And this package is there as well

>pip list

Package                           Version
--------------------------------- -------
attrs                             21.2.0
.
.
opc                               0.8.0

Interestingly, when I run the python code directly like

python3 scripts/download.py -d dir_path

it works fine. I don't not get ModuleNotFoundError error.

Also, this works fine as well:

❯ python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import opc.utils.global_vars as g
>>> print(g.LOCAL_HOST)
127.0.0.1
>>>

Directory structure:

/home/ray/Desktop/repo/image-builder/scripts/download.py
/home/ray/Desktop/repo/image-builder/scripts/src/build.sh

and i am running commands from:

/home/ray/Desktop/repo/image-builder

Answer 1

This is likely to be because when you invoke the shell script, the python path is different from when you're running the Python interpreter. If that is the case, there are several ways to address that.

What happens if you manually change into the same directory as download.py and run your python script from there? Something like:

cd scripts
python download.py -d <some-dir>

Does that work, or do you still hit the same problem?

In shell scripts like this I first make sure the script will always be invoking the python app from the exact same place in the file system, so that I don't hit these python path issues. If you know for sure that python download.py -d <some-dir> works from inside the same directory as download.py , then you can add something like this to your shell script (assuming your shell script and download.py are in the same directory):

pushd "${0%/*}" # this changes directory to the directory where the shell script itself lives

# do your stuff here - run your python app, etc.

popd # returns to whatever directory we were originally in

In this way, I can always invoke the shell script from whatever directory I am in and I know it will behave the same way - the current directory is no longer relevant.

I wouldn't use $(pwd) , because that will give the current directory, which varies, so your shell script will only ever work properly when invoked from inside the right directory if you do that.