Run celery worker with a compiled python module compiled using pyinstaller

Question

I have a simple celery module ( worker.py ) that looks like:

from celery import Celery

app = Celery('celery')

@app.task(bind=True)
def some_task(self):
    print('HELLO WORLD')

I'm compiling the module using pyinstaller :

python3 -m PyInstaller --onefile worker.py

Then, I can run a celery worker with the python module quite easily using the command:

celery -A worker.app worker -l info

My question is: How can I run the celery worker with the compiled version of the module (Created by PyInstaller)?

Answer 1

I don't think this is possible (but please do correct me if it is).

Firstly, pyinstaller generates an executable and thus a file that comes from the output of pyinstaller is a hardcoded bin and can run perfectly without an interpreter, dependencies, etc.

From the pypi

PyInstaller bundles a Python application and all its dependencies into a single package. The user can run the packaged app without installing a Python interpreter or any modules.

Celery requires the use of a python interpreter to run the python code, ie, Celery is a simple library that is imported and integrates with the existing python code to run the file. Thus, an executable file that is compiled with pyinstaller shouldn't work in this scenario.

Answer 2

The solution for this problem is to run the worker from inside a python module, so for example:

from celery import Celery

app = Celery('celery', fixups=[])

@app.task(bind=True)
def some_task(self):
    print('HELLO WORLD')

if __name__ == "__main__":
    app.worker_main(argv=['worker', '--loglevel=info'])

and then we can use pyinstaller to compile this module:

pyinstaller --one-file module.py --additional-hooks-dir=pyinstaller_hooks_folder

the additional-hooks-dir flag is for pyinstaller to know how to collect celery's submodules, etc.. we need to have pyinstaller_hooks_folder in our project folder and a celery hook file in it:

Project_folder
└── pyinstaller_hooks_folder
    └──hook-celery.py 

hook-celery.py file:

from PyInstaller.utils.hooks import collect_all

datas, binaries, hiddenimports = collect_all('celery')