python import of local module failing when run as systemd/systemctl service

Question

I have a python application that I'm tring to run as a system service. The application runs fine when I run it manually. When I run it as a service it fails to find a local module that was installed with pip install -e my_module .

The main of the application has the following code:

print(sys.argv)
import pip
installed_packages = pip.get_installed_distributions()
installed_packages_list = sorted(["%s==%s" % (i.key, i.version) for i in installed_packages])
print(installed_packages_list)
print('doing tox')
import tox
print('doing my_mod')
import my_mod
print(my_mod.__file__)
from my_mod.auth.http_auth_provider import HTTPAuthProvider

When I run it manually I get (note that my-mod is include on second line in 'installed packages'):

['/usr/bin/pv_api']
['aiohttp==0.19.0', 'chardet==2.3.0', 'jsonschema==2.5.1', 'pip==7.0.0', 'pluggy==0.3.1', 'pv-api==0.0.0', 'py==1.4.31', 'pycrypto==2.6.1', 'pymongo==3.1.1', 'pyyaml==3.11', 'setuptools==19.6.2', 'six==1.10.0', 'tox==2.3.1', 'virtualenv==14.0.6', 'my-mod==0.1.0', 'webauthsession==1.1.1']
doing tox
doing my_mod
/root/my_module/my_mod/__init__.py

When run through the service the logs look like this (note that my-mod is NOT included on second line in 'installed packages')::

2016-02-26_00:39:01.90403 ['/usr/bin/pv_api']
2016-02-26_00:39:01.90406 ['aiohttp==0.19.0', 'chardet==2.3.0', 'jsonschema==2.5.1', 'pip==7.0.0', 'pluggy==0.3.1', 'pv-api==0.0.0', 'py==1.4.31', 'pycrypto==2.6.1', 'pymongo==3.1.1', 'pyyaml==3.11', 'setuptools==19.6.2', 'six==1.10.0', 'tox==2.3.1', 'virtualenv==14.0.6', 'webauthsession==1.1.1']
2016-02-26_00:39:01.90407 doing tox
2016-02-26_00:39:01.90407 doing my_mod
2016-02-26_00:39:01.90642 Traceback (most recent call last):
2016-02-26_00:39:01.90642   File "/usr/bin/pv_api", line 9, in <module>
2016-02-26_00:39:01.90642     load_entry_point('pv-api==0.0.0', 'console_scripts', 'pv_api')()
2016-02-26_00:39:01.90643   File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 547, in load_entry_point
2016-02-26_00:39:01.90643     return get_distribution(dist).load_entry_point(group, name)
2016-02-26_00:39:01.90643   File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2719, in load_entry_point
2016-02-26_00:39:01.90643     return ep.load()
2016-02-26_00:39:01.90643   File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2379, in load
2016-02-26_00:39:01.90643     return self.resolve()
2016-02-26_00:39:01.90643   File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2385, in resolve
2016-02-26_00:39:01.90644     module = __import__(self.module_name, fromlist=['__name__'], level=0)
2016-02-26_00:39:01.90644   File "/usr/lib/python3.4/site-packages/pv/api/main.py", line 33, in <module>
2016-02-26_00:39:01.90644     import my_mod
2016-02-26_00:39:01.90644 ImportError: No module named 'my_mod'

This might also be useful information:

[root@7bb8a6866a85 etc]# ls -la /usr/lib/python3.4/site-packages/my-mod.egg-link 
-rw-r--r-- 1 root root 37 Feb 26 00:20 /usr/lib/python3.4/site-packages/my-mod.egg-link
[root@7bb8a6866a85 etc]# cat /usr/lib/python3.4/site-packages/my-mod.egg-link 
/root/my_module

Edit:

As you can see from the output of 'installed_packages' all other packages that are installed via requirements.txt are found correctly. Just this one library that I have source code for locally is not found when I run as service. (It is found when I run from the command line or when I run import my_mod from the python3 interpreter.

Answer 1

I had a very similar issue converting an upstart heartbeat.conf to a systemd heartbeat.service , except with the requests module. The solution was to specify in the new .service what user to run it as:

[Unit]
Description=web server monitor

[Service]
WorkingDirectory=/home/user/
User=user
ExecStart=/home/user/heartbeat.py
Restart=always

[Install]
WantedBy=multi-user.target

Without the User=user , I was getting in the journalctl:

systemd[1]: Started web server monitor.
heartbeat.py[26298]: Traceback (most recent call last):
heartbeat.py[26298]:   File "/home/user/heartbeat.py", line 2, in <
heartbeat.py[26298]:     import requests
heartbeat.py[26298]: ImportError: No module named requests
systemd[1]: heartbeat.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: heartbeat.service: Unit entered failed state.

Answer 2

First try the following in python prompt.

$ python
>>> import my_mod
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named my_mod
>>>

Fix 1

If you are getting the above sort of output then the cause may be because of permission issue. Grant permission for site-packages using the following.

sudo chmod -R go+rX /usr/local/lib/python2.7/dist-packages

Fix 2

Try exporting the PYTHONPATH as below:

export PYTHONPATH="/usr/.local/lib/python2.7/site-packages"

Fix 3

Check if you have multiple version of python running in same machine.

If so, check whether you have proper interpreter is included at the beginning of the code like #!/usr/bin/python

Answer 3

如果您确实想以 root 身份运行该服务，则必须使用 sudo 安装模块： sudo pip install my_module 。

Answer 4

1) Install the supervisor package ( more verbose instructions here ):

sudo apt-get install supervisor

2) Create a config file for your daemon at /etc/supervisor/conf.d/my_mod.conf :

[program:my_mod]
directory=/path/to/project/root
environment=ENV_VARIABLE=example,OTHER_ENV_VARIABLE=example2
command=python my_mod.py
autostart=true
autorestart=true

3) Restart supervisor to load your new .conf

supervisorctl update
supervisorctl restart my_mod

Answer 5

Since your python script is executing with python3.4 , your problem is very likely the use of pip instead of pip3 (or possibly sudo -H pip3 ).

But since you also ask specifically about using python modules from a systemd service, I would recommend using apt with sudo if at all possible, instead of messing with pip (which is notorious for introducing problems with file permissions):

sudo apt install python3-my_module

Answer 6

Add the python site packages environment variable to the systemctl *. Service file

[Unit]
Description=web server monitor

[Service]
WorkingDirectory=/home/user/
User=user
ExecStart=/home/user/heartbeat.py
Restart=always
Environment="PYTHONPATH=$PYTHONPATH:/home/nvidia/.local/lib/python3.6/site-packages"

[Install]
WantedBy=multi-user.target

Answer 7

I'm also getting import error but mine was not a local import. My package was not imported correctly. See service failure logs*

Feb 11 06:41:52 pl-dev-demo-1 python3[1675804]:     from foo import test
Feb 11 06:41:52 pl-dev-demo-1 python3[1675804]: ModuleNotFoundError: No module named 'foo'
Feb 11 06:41:52 pl-dev-demo-1 systemd[1]: xyzxyz.service: Main process exited, code=exited, status=1/FAILURE

I wasted a lot of time to figure out then I tried this approach which worked perfectly.

In module I set the path of my code :

import sys
sys.path.append('path_of_my_code_parent_package')

from foo import test 

Read this doc :

Setting PYTHONPATH more permanently

Answer 8

I had the same problem. I figured pip install must be user specific.
So I switched to root and then installed the packages. It worked after that.

However, I think specifying User=myUser in the service file would be a more proper way, however, I wanted it to run with root permissions and I wasn't sure it will when I specify the user.

Hope it helps someone