(java) HttpEntity.getContent() outputs limited InputStream (8192b / 7748b) … I'm trying to download a 2.4MB file

Question

I'm using Apache HttpClient 4.5 , trying to download a 2.4 MB file . File is available and fully downloadable in Chrome.

Problem is, InputStream is returned by entity.getContent() contains only 8192 bytes of buffer, and is.available() returns 7748. Private filed contentLength of is is 2488649, which should be the desired file size, so I don't understand what the problem is. All buffer sizes that I found in Variables module of Debug are 8192.

I have tried to replace the is on-the-spot with FileInputStream targeting the same file on my computer. It transferred perfectly, all 2.4 MB. So I suppose I'm doing something wrong with http utils configuration.

Please help me fix this. The code seems a bit long but it should be easy to comprehend.

CommRunnable. Base class for the class below:

public abstract class CommRunnable implements Runnable {
    protected HttpPostAgent agent;
    protected boolean success;

    //...additional methods...//
}

GetFileRunnable. Running in separate thread. Initiates the connection and the file transfer:

public class GetFileRunnable extends CommRunnable implements Runnable {
    private String url;
    private String fileDestination;
    private NameValuePair[] postPairs;

    //...constructor...//

    public void run() 
    {
        synchronized (agent) {
            try {
                HttpClient client = HttpClients.createDefault();

                HttpResponse response = agent.getHttpResponse(client, url, postPairs);

                InputStream is = response.getEntity().getContent();
                FileOutputStream fos = new FileOutputStream(fileDestination);

                success = agent.transfer(is, fos);

                client.getConnectionManager().shutdown();

            } catch (IOException e) {
                e.printStackTrace();
                throw new RuntimeException("IOException in HttpPostAgent.getFile");
            } finally {
                agent.setFree(true);
            }
        }
    }
}

HttpPostAgent. Basically a controller for Apache http utilities. Methods are described in comments above them.

public class HttpPostAgent {
    private int statusPerc;
    private boolean free;

    private Thread thread;
    private CommRunnable currentAction;

    //...constructor...//

    //executes POST request and returns resulting HttpResponse
    HttpResponse getHttpResponse(HttpClient client, String url, NameValuePair... postPairs)
    {
        try {
            List <NameValuePair> nvps = new ArrayList <NameValuePair>();

            for (NameValuePair pair : postPairs)
                nvps.add(pair);

            HttpPost post = new HttpPost(url);
            post.setEntity(new UrlEncodedFormEntity(nvps, HTTP.UTF_8));

            return client.execute(post);

        } catch (IOException e) {
            throw new RuntimeException("IOException in getHttpResponse(HttpClient client, String url, NameValuePair... postPairs)");
        }
    }

    //...methods...//

    //Starts thread with GetFileRunnable. Included here for Your understanding of my context.
    public void getFileInit(String url, String destinationPath, NameValuePair... postPairs)
    {
        free = false;
        statusPerc = 0;
        currentAction = new GetFileRunnable(this, url, destinationPath, postPairs);
        thread = new Thread(currentAction);
        thread.setPriority(Thread.MIN_PRIORITY);
        thread.setDaemon(true);
        thread.start();
    }

    //...methods...//

    //Transfers a file from one input stream to the other. For downloading from response/entity/content.
    boolean transfer(InputStream is, OutputStream os)
    {
        boolean success = false;

        try {
            int remain = is.available();
            int total = remain;
            int read = 0;
            int chunk = 1024;
            byte[] buffer = new byte[chunk];

            while(remain > 0)
            {
                if(chunk > remain) chunk = remain;

                os.flush();
                is.read(buffer, 0, chunk);
                os.write(buffer, 0, chunk);

                remain -= chunk;
                read += chunk;

                synchronized (this) {
                    statusPerc = (read * 100) / total;
                }
            }
            remain = is.available();

            success = true;

        } catch (IOException e) {
            throw new RuntimeException("IOException in HttpPostAgent.transfer");
        } 

        return success;
    }

    //...methods...//
}

Please tell me if I should add any more information.

Answer 1

From the Javadoc of InputStream.available() :

Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream.

So the available bytes are not the total input bytes, but the first chunk of bytes which have been buffered when you start reading the input. 8192 seems to be the buffer size in your example.

Therefore your HttpPostAgent.transfer method effectively only handles the first 8192 bytes and then stops.

Can you try the following replacement for the HttpPostAgent.transfer method?

boolean transfer(InputStream is, OutputStream os) throws IOException
{
    byte buffer[] = new byte[2048];
    int count;
    while ((count = is.read(buffer)) != -1)
        os.write(buffer, 0, count);
}