Serving Chunked Transfer Coding from flask through nginx

Question

I have some issues serving Chunked Transfer Coding through Nginx.

The data served comes from a flask app running through uwsgi. Here is a sample response that uwsgi gives to a GET request:

0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a                                              .
<= Recv header, 28 bytes (0x1c)
0000: 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e Transfer-Encodin
0010: 67 3a 20 63 68 75 6e 6b 65 64 0d 0a             g: chunked..
<= Recv header, 32 bytes (0x20)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
0010: 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 0d 0a plication/json..
<= Recv header, 32 bytes (0x20)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 41 Access-Control-A
0010: 6c 6c 6f 77 2d 4f 72 69 67 69 6e 3a 20 2a 0d 0a llow-Origin: *..
<= Recv header, 75 bytes (0x4b)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 41 Access-Control-A
0010: 6c 6c 6f 77 2d 48 65 61 64 65 72 73 3a 20 41 75 llow-Headers: Au
0020: 74 68 6f 72 69 7a 61 74 69 6f 6e 2c 20 43 6f 6e thorization, Con
0030: 74 65 6e 74 2d 54 79 70 65 2c 20 43 6f 6e 74 65 tent-Type, Conte
0040: 6e 74 2d 4c 65 6e 67 74 68 0d 0a                nt-Length..
<= Recv header, 80 bytes (0x50)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 45 Access-Control-E
0010: 78 70 6f 73 65 2d 48 65 61 64 65 72 73 3a 20 20 xpose-Headers:  
0020: 57 57 57 2d 41 75 74 68 65 6e 74 69 63 61 74 65 WWW-Authenticate
0030: 2c 20 43 6f 6e 74 65 6e 74 2d 54 79 70 65 2c 20 , Content-Type, 
0040: 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 0d 0a Content-Length..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a                                           ..
<= Recv data, 17 bytes (0x11)
0000: 63 0d 0a 7b 22 64 65 76 69 63 65 22 3a 20 5b 0d c..{"device": [.
...DATA...
01a0: 32 0d 0a 5d 0a 0d 0a 32 0d 0a 7d 0a 0d 0a       2..]...2..}...
<= Recv data, 5 bytes (0x5)
0000: 30 0d 0a 0d 0a                                  0....

This response then goes into a unix socket which is piped into nginx with the following configuration:

server {
listen      80;
server_name  _;
client_max_body_size 4m;
chunked_transfer_encoding on;

location = /abc { rewrite ^ /abc/; }
location /abc { try_files $uri @abc; }
location @abc {
    include uwsgi_params;
uwsgi_param SCRIPT_NAME /abc;
    uwsgi_modifier1 30;
    uwsgi_pass unix:/path/to/socket.sock;
}

The problem is that Nginx actually gives this response:

<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a                                              .
== Info: Server nginx/1.4.6 (Ubuntu) is not blacklisted
<= Recv header, 30 bytes (0x1e)
0000: 53 65 72 76 65 72 3a 20 6e 67 69 6e 78 2f 31 2e Server: nginx/1.
0010: 34 2e 36 20 28 55 62 75 6e 74 75 29 0d 0a       4.6 (Ubuntu)..
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 54 75 65 2c 20 30 38 20 4a 75 Date: Tue, 08 Ju
0010: 6c 20 32 30 31 34 20 30 31 3a 30 32 3a 33 33 20 l 2014 01:02:33 
0020: 47 4d 54 0d 0a                                  GMT..
<= Recv header, 32 bytes (0x20)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
0010: 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 0d 0a plication/json..
<= Recv header, 28 bytes (0x1c)
0000: 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e Transfer-Encodin
0010: 67 3a 20 63 68 75 6e 6b 65 64 0d 0a             g: chunked..
<= Recv header, 24 bytes (0x18)
0000: 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 6b 65 65 70 Connection: keep
0010: 2d 61 6c 69 76 65 0d 0a                         -alive..
<= Recv header, 32 bytes (0x20)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 41 Access-Control-A
0010: 6c 6c 6f 77 2d 4f 72 69 67 69 6e 3a 20 2a 0d 0a llow-Origin: *..
<= Recv header, 75 bytes (0x4b)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 41 Access-Control-A
0010: 6c 6c 6f 77 2d 48 65 61 64 65 72 73 3a 20 41 75 llow-Headers: Au
0020: 74 68 6f 72 69 7a 61 74 69 6f 6e 2c 20 43 6f 6e thorization, Con
0030: 74 65 6e 74 2d 54 79 70 65 2c 20 43 6f 6e 74 65 tent-Type, Conte
0040: 6e 74 2d 4c 65 6e 67 74 68 0d 0a                nt-Length..
<= Recv header, 79 bytes (0x4f)
0000: 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d 45 Access-Control-E
0010: 78 70 6f 73 65 2d 48 65 61 64 65 72 73 3a 20 57 xpose-Headers: W
0020: 57 57 2d 41 75 74 68 65 6e 74 69 63 61 74 65 2c WW-Authenticate,
0030: 20 43 6f 6e 74 65 6e 74 2d 54 79 70 65 2c 20 43  Content-Type, C
0040: 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 0d 0a    ontent-Length..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a                                           ..
<= Recv data, 926 bytes (0x39e)
0000: 33 39 32 0d 0a 63 0d 0a 7b 22 64 65 76 69 63 65 392..c..{"device
... DATA ...
0380: 0a 7d 0d 0a 32 0d 0a 5d 0a 0d 0a 32 0d 0a 7d 0a .}..2..]...2..}.
0390: 0d 0a 30 0d 0a 0d 0a 0d 0a 30 0d 0a 0d 0a       ..0......0....

So it basically puts all the chunks generated by the flask app and put them in one big chunk that he is serving in chunked transfer coding.

All the other functionality (basic HTTP 1.0) are perfectly working so I wonder what could be the reason for this behaviour.

tl;dr : NGINX converts a list of chunks into one big, and leaves the hex digits in the middle of the json data served

Answer 1

Apparently, This problem is the similar to the one I got.

The solution I found is to enable transfer encoding on nginx

chunked_transfer_encoding on;

And then to stream the packets to nginx one by one: It is transfer-encoding chunked without the hex size markers and the CRLFs.

What happens (I think) is that nginx buffers the data on its side (through the socket it is listening to). This way it prevents you from overflowing your memory usage on your the python/java side (I won't settle for Python's or Java's memory efficiency).

I look forward to any better solution, but for not so big/numerous queries (<50Mb total) it works on small instances. I'll keep this page updated with new findings.