Multiprocessing with in a linux daemon written in python

Question

I have a linux daemon (based on python module python-daemon) that needs to spawn two processes (consider a producer and a consumer) of the Multiprocessing module to handle some concurrent I/O (the producer reads from an input stream and the consumer uploads the data using python requests). As per the python docs ( https://docs.python.org/2/library/multiprocessing.html ), daemonic processes are not allowed to start child processes. How can I handle this? Are there any documents or examples for this approach? Pls. advise.

Context: I have tried using threading module. But, due to the GIL problem, the consumer rarely gets a chance to execute. I also looked into tornado and gevent. But, that would require rewriting a lot of the code.

Answer 1

I think there is some confusion here. Document says only if you mark the process that has been created from python as daemon then it cannot create sub process. But your python-daemon is a normal linux daemon.

linux daemon - process running in background. (python daemon library creates such process), these can have subprocess

Only a daemon process created from multiprocessing library cannot create sub-process.