How to received chunked responses in HTTP/1.1 while sending data at the same time to server in Java/Android

Question

We're building and app on which it records the voice of the user in realtime and sends the recorded data to the server through an HTTP request. While the server processes the data in realtime it is also sending back responses in chunked. To put it simple, the app is sending data piece by piece to the server, and at the same time, it is also receiving piece by piece responses from the server.

Please DON'T tell me this is impossible because I have a working example in iOS which uses the URLSession with uploadTask using a stream pair to send the data in real time to the server and then receiving responses chunk by chunk from this callback urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) .

Below is my code in Java. I got the sending working, but the response is only received when the sending is done.

RequestBody body = new RequestBody() 
{
    @Override 
    public MediaType contentType() 
    {
        return MediaType.get("application/octet-stream");
    }

    @Override 
    public void writeTo(BufferedSink sink) throws IOException 
    {
        String filename = "/path/spoken.pcm";

        try 
        {
            InputStream inputStream = new DataInputStream(new FileInputStream(new File(filename)));

            byte[] cacheBytes = new byte[320];

            int length;

            while((length = inputStream.read(cacheBytes, 0, cacheBytes.length)) != -1) 
            {
                System.out.println("write thread name: " + Thread.currentThread());

                sink.write(cacheBytes, 0, length);

                Thread.sleep(10);
            }

            inputStream.close();
        } 

        catch(IOException | InterruptedException e) 
        {
            e.printStackTrace();
        }
    }
};

Request request = new Request.Builder()
            .url("www.server.com")
            .post(body)
            .build();

Interceptor interceptor = new Interceptor()
{
    @Override
    public Response intercept(Chain chain) throws IOException 
    {
        System.out.println("intercept!!!");

        CountDownLatch latch = new CountDownLatch(1);

        Response response = chain.proceed(chain.request());
        BufferedSource source = response.body().source();

        System.out.println("got response body !!!!");

        new Thread(new Runnable()
        {
            @Override
            public void run() 
            {
                byte[] cachedBytes = new byte[512];

                try 
                {
                    while(!source.exhausted())
                    {
                        int length = source.read(cachedBytes);
                        byte[] partialBytes = new byte[length];

                        System.arraycopy(cachedBytes, 0, partialBytes, 0, length);

                        System.out.println("partial response received: " + getHexString(partialBytes));
                    }
                } 

                catch (IOException e) 
                {
                    e.printStackTrace();
                }

                latch.countDown();
            }
        }).start();

        try 
        {
            latch.await();
        } 

        catch (InterruptedException e) 
        {
            e.printStackTrace();
        }

        return response;
    }
};

httpClient = new OkHttpClient.Builder()
            .addInterceptor(interceptor)
            .build();

httpClient.newCall(request).enqueue(new Callback() 
{
    @Override
    public void onFailure(Call call, IOException e) 
    {
        e.printStackTrace();
    }

    @Override
    public void onResponse(Call call, Response response) throws IOException 
    {
        try(ResponseBody responseBody = response.body()) 
        {
            if(!response.isSuccessful()) throw new IOException("Unexpected code " + response);

            System.out.println("all responses received!");
        }
    }
});

This log: System.out.println("got response body !!!!"); appears only when I am done sending all the data to server. Which means, when writeTo(BufferedSink sink) returns, I get the response within the interceptor callback in chunks and then the callback onResponse(Call call, Response response) gets called.

What I need is, when I am sending data, I want to be able to get the chunked responses at the same time.

Answer 1

For HTTP/1.1, OkHttp won't return the response body until the request body has completed transmitting. For HTTP/2 you can override isDuplex() to return true.

Answer 2

OK, by looking up everywhere, this implementation is not the standard way as others say for HTTP. Although HTTP does not limit you from doing a send and receive simultaneously in a single request, as I have achieved this already using an iOS App as client and with a Java HTTP Servlet as the server. The problem is, most libraries does not support this behavior. You may write your own SDK for this on client side, but you should also consider the risk in the future if you plan to use Load Balancers and Reverse Proxies for your server, and they might not support this behavior either.

So, my quickest solution to achieve the same effect is using HTTP for sending the request, but the response would be sent to the client through MQTT.

Answer 3

You could try implementing a socket that will stay open and keep listening until the app is closed. This way responses from your server can pour in and you just have to parse it and grab what you need. Java Socket will allow you to do such implementation.Java socket allows you to independently handle input stream and output stream. In your case, the input stream will stay opened forever.

The problem with your above solution is that you are sending a request to the server. The request is completed through a callback once the server replies (200 OK,401 ...etc).Keep in mind the later mechanic is asynchronous and stateless.