为Python脚本创建共享消息流的最佳方法是什么？

Question

What I want to do: I need a simple message stream, so some scripts can send results there and another script can take results and do some work asynchronously.

Main problem: I want to see what's happening, so if something breaks - I can fix it quickly. I tried to use Celery+RabbitMQ (can see workers with args, using Flower, but scheduling too complicated) and multiprocessing.Queue (simple, but can't see workers with args).

---

What I've done: I tried to build something similar, using MongoDB capped collection and run Popen with multiple processes, to react. Some scripts write smth to the collection, the script below monitors it and if some condition is met - run another script.

Main problem: subprocess.Popen() usage from inside multiprocessing.Process() looks unnatural (still does the work), so I'm trying to find better or/and more stable solution :)

---

Listener script:

from pymongo import MongoClient, CursorType
from time import sleep
from datetime import datetime

from multiprocessing import Process
import subprocess

def worker_email(keyword):
     subprocess.Popen(["python", "worker_email.py", str(keyword)])

def worker_checker(keyword):
     subprocess.Popen(["python", "worker_checker.py", str(keyword)])

if __name__ == '__main__':

    #DB connect
    client = MongoClient('mongodb://localhost:27017/')
    db = client.admetric
    coll = db.my_collection
    cursor = coll.find(cursor_type = CursorType.TAILABLE_AWAIT)

    #Script start UTC time
    utc_run = datetime.utcnow()

    while cursor.alive:
        try:
            doc = cursor.next()
            #Print doc name/args to see in command line, while Listener runs
            print(doc)
            #Filter docs without 'created' data
            if 'created' in doc.keys():
                #Ignore docs older than script
                if doc['created'] > utc_run:
                    #Filter docs without 'type' data
                    if 'type' in doc.keys():
                        #Check type
                        if doc['type'] == 'send_email':
                            #Create process and run external script
                            p = Process(target=worker_email, args=(doc['message'],))
                            p.start()
                            p.join()
                        #Check type
                        elif doc['type'] == 'check_data':
                            #Create process and run external script
                            p = Process(target=worker_checker, args=(doc['message'],))
                            p.start()
                            p.join()
        except StopIteration:
            sleep(1)

Answer 1

As long as you have control over the worker_email and worker_checker logic, you don't need to execute the in a separate interpreter.

Just expose an entry point in the two modules and run them via multiprocessing.Process .

worker_email.py

def email_job(message):
    # start processing the message here

worker_checker.py

def check_job(message):
    # start checking the message here

listener_script.py

# you are not going to pollute the listener namespace
# as the only names you import are the entry points of the scripts
# therefore, encapsulation is preserved
from worker_email import email_job
from worker_checker import check_job

email_process = Process(target=email_job, args=[message])
check_process = Process(target=check_job, args=[message])

If you cannot expose an entry point from the worker modules then just run subprocess.Popen . You have no benefit in wrapping them within a Process .