How to scale a CPU bound Twisted application?

Question

I'm working on a twisted web application which uploads files and encrypts them, returning the url+key to the user.

I've been tasked with scaling this application. At the moment when there are more than 3-4 concurrent upload requests the performance will drop off significantly.

I'm no Twisted expert but I assume this is due to it running in a single python process, being a high cpu application and the GIL?

How could I go about scaling this?

If this was a different framework such as Flask I would just put uwsgi in front of it and scale the number of processes. Would something similar work for Twisted and if so what tools are generally used for this?

Answer 1

If you think you could throw uwsgi in front of the application, I suppose it is pretty close to shared-nothing. So you can run multiple instances of the program and gain a core's worth of performance from each.

There are a couple really obvious options for exactly how to run the multiple instances. You could have a load balancer in front. You could have the processes share a listening port. There are probably more possibilities, too.

Since your protocol seems to be HTTP, any old HTTP load balancer should be applicable. It needn't be Twisted or Python based itself (though certainly it could be).

If you'd rather share a listening port, Twisted has APIs for passing file descriptors between processes ( IReactorSocket ) and for launching new processes that inherit a file descriptor from the parent ( IReactorProcess ).