PHP push to message queue; process with multithreaded python client?

Question

Here's what I'm trying to do: use PHP to push a message into a queue (either Beanstalkd , IronMQ , Amazon SQS ).

Then I want a Python script to instantly pick up the message. I guess I'll just have 1 Python script running a while(true) loop to keep polling the message server?

Then process it using a new thread for each job. If there's 10 messages in the queue, I want python to run 10 threads, 1 for each job.

My questions are:

1. Is this a solid way of doing things, or is there a better way to set this up?
2. How do I ensure my Python script is up and polling forever?

Answer 1

Is this a solid way of doing things, or is there a better way to set this up?

Sounds reasonable to me, although if you're expecting a large number of simultaneous jobs, you may want to limit the total number of threads by using a thread pool .

You won't gain much more CPU performance, for CPU-intensive threads, when the total number of threads exceeds the total number of CPU cores, and if the threads require significant disk I/O, you'll want to limit them to avoid thrashing the disks.

How do I ensure my Python script is up and polling forever?

It's common for daemon processes to have another process which monitors them, and restarts them if they crash or become unresponsive. A simple cronjob might suffice for this purpose.