twisted python hangs on large multipart messages

Question

At my company we are using twisted and flask to run a rest api service on an industrial computer. This API is used partially for IPC and for limited communication from external sources. We've recently found a problem where twisted hangs when sending files via multipart post. Files around 600MB will hang twisted for about 40-50 seconds, no requests are processed during this time.

I should say I am not a web developer, I'm learning as I go. I've analyzed the issue and isolated it to the assembly of the multipart message, flask hangs on this as well but doesn't block messages. There have been a couple of bug reports for the same/similar issue: https://twistedmatrix.com/trac/ticket/5511 . Problem is these are from 8 years ago, and I'm not concerned with streaming, or at least I don't think I am.

Are just out of luck? Is there anything we can do to twisted for this work, I thought perhaps a configuration issue. Or perhaps an alternative to twisted, oddly enough the WSGI that comes with flask doesn't have this problem, but it is not for production use so it's not an option.

Answer 1

Apart from implementing a fix from the problem, you may indeed be out of luck. The reason you notice this problem with Twisted is that your Twisted Web server is almost certainly entirely single threaded (this is the default and doing otherwise requires some non-trivial effort). When the single Twisted reactor thread starts parsing the huge upload, nothing else can be served until it has finished parsing that huge upload. And as you've discovered, the parser is quite slow.

You don't observe this problem (at least not under the same conditions) with a WSGI-based server because those servers are running multiple threads or processes. When one client uploads a large file, one of those threads or processes is probably blocked for a while but the others can continue to service other requests. If you had one client per thread or process and those clients all uploaded a large file, you would still block all of the threads/processes and other clients would not be serviced until those threads/processes finished that work.

As I said at the top, this is not impossible to fix. It can be fixed in Twisted so that form parsing does not block the reactor thread until it has finished completely. There are likely other solutions as well. For example, you may be able to run multiple Twisted Web processes (perhaps sharing a single listening socket to make this transparent to clients) so that it at least behaves as well as the WSGI server you observed.