Django with mod_wsgi and gzip

Question

I am using Django as a rest server. I suppose to get a POST that contains JSON that I should parse. The client is a salesforce server that is gzipping the request.

To get the request inflated, I use this in VHost: SetInputFilter DEFLATE

Almost everything looks fine, but when I read request.body or request.read(16000) - input is pretty small - I always see chopped response (5 characters are missing).

Any suggestions where to start debugging?

Answer 1

Technically the WSGI specification doesn't support the concept of mutating input filters as middleware, or even within a underlying web server.

The specific issue is that mutating input filters will change the amount of request content, but will not change the CONTENT_LENGTH value in the WSGI environ dictionary.

The WSGI specification says that a valid WSGI application is only allowed to read up to CONTENT_LENGTH bytes from the request content. As a consequence, in the case of compressed request content, where the final request size will end up being greater that what CONTENT_LENGTH specifies, a web framework is likely to truncate the request input before all data is read.

You can find some details about this issue in:

• http://blog.dscpl.com.au/2009/10/details-on-wsgi-10-amendmentsclarificat.html

Although changes in the specification were pushed for, nothing ever happened.

To work around the problem, what you would need to do is implement a WSGI middleware which you would wrap around the Django application, which if it detects by way of headers passed, that the original content had been compressed, but where you know Apache decompressed it, would read all request content until it reach the end of stream marker, ignoring CONTENT_LENGTH, before even passing the request to Django. Having done that, it could then change CONTENT_LENGTH and substitute wsgi.input with a replacement stream which then returns the already read content.

Because the content size could be quite large and of unknown size, reading it all into memory would not necessarily be a good idea. You therefore would likely want to read it in a block at a time and write it out to a temporary file. The wsgi.input would then be replaced with an open file handle on the temporary file and CONTENT_LENGTH replaced with the final size of the file.

If you search properly on the mod_wsgi archives on Google Groups, you should find prior discussions on this and perhaps even some example code.