Twisted with queue for CPU-bound tasks

Question

I have an HTTP server which does some IO stuff, then does some CPU-bound stuff (PIL) and then replies with data (magnitude of megabyte or so).

(a) My first idea is something like this: a process for server and IO, based on Twisted, and several processes for PIL stuff, with queue.

If this architecture is reasonable, then there probably is a library which does exactly that: multiprocess queue for Twisted. However, I'm not really experienced in Twisted and know nothing of its community so the only thing I found is ampoule , for which I found neither docs nor description which would persuade me that it's the right tool for a job.

(b) Another idea is to just run several servers in several threads, with both IO and CPU stuff going in each on of them. This seems stupid because CPU stuff will block, but maybe I'm not really understanding it.

So, questions:

1. Is any of these architectures reasonable?
2. How would you implement it (using Twisted + ampoule or what?)
3. For (a), how would you send a huge pile of data from "worker" to the server thread? Or maybe I can tell the worker to write into the response directly somehow?
4. How many "workers" are reasonable?

Answer 1

1. Yes, those architectures are probably reasonable
2. I'd probably use ampoule too, but I don't know a lot about it right now either. This link is the closest thing to a good introduction that I remember seeing.
3. Since it sounds like your workers will always be on the same machine, you should be able to use shared memory.
4. That depends pretty wildly on the number of cores you have available and how intensive the work is (both with respect to CPU time and other resources, like memory and disk). It's probably hard to give any answer better than "benchmark with around 1-5 processes per core and see what's fastest".