How to get celery to work with SCL python and systemd?

Question

I am having a Python application that uses celery for background tasks. I use Python interpreter provided by SCL.

I was able to create a systemd unit file for teh app: How to use user's pipenv via systemd? Python is installed via SCL

But I do not understand how to put a similar sytstemd unit file for celery.

I tried:

[Unit]
Description=app celery service

# Requirements
Requires=network.target

# Dependency ordering
After=network.target

[Service]
# Let processes take awhile to start up
TimeoutStartSec=0
Type=forking

RestartSec=10
Restart=always

Environment="APP_SITE_SETTINGS=/home/app/.config/settings.cfg"
Environment="PYTHONPATH=/home/app/.local/lib/python3.6/site-packages"
WorkingDirectory=/home/app/app-site/app

User=app
Group=app
PermissionsStartOnly=true

KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

# Main process
ExecStart=/usr/bin/scl enable rh-python36 -- /home/app/.local/bin/pipenv run celery -A celery_service.celery worker

[Install]
WantedBy=multi-user.target

When I start the systemd unit, in the journal I see that the celery app starts. After a few seconds, the service fails.

Job for app_celery.service failed because a timeout was exceeded. See "systemctl status app_celery.service" and "journalctl -xe" for details.

Here's a journal entry:

Jul 17 07:43:31 some.host.com scl[5181]: worker: Cold shutdown (MainProcess)

I tried with Type=oneshot and Type=simple too. None of them worked. I suspect this has something to do with SCL.

Is there a way to get the celery app to work with SCL and systemd?

Answer 1

Celery has a command line option --detach. When you use --detach Celery starts workers as a background process.

Here's the working systemd unit file:

[Unit]
Description=app celery service

# Requirements
Requires=network.target

# Dependency ordering
After=network.target

[Service]
# Let processes take awhile to start up
TimeoutStartSec=0
Type=simple
RemainAfterExit=yes
RestartSec=10
Restart=always
Environment="SETTINGS=/home/app/.config/settings.cfg"
Environment="PYTHONPATH=/home/app/.local/lib/python3.6/site-packages"
WorkingDirectory=/home/app/app-site/app/

User=app
Group=app
PermissionsStartOnly=true

KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

LimitMEMLOCK=infinity
LimitNOFILE=20480
LimitNPROC=8192

# Main process
ExecStart=/usr/bin/scl enable rh-python36 -- /home/app/.local/bin/pipenv run celery -A celery_service.celery worker --detach

[Install]
WantedBy=multi-user.target