Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection WARNING

Question

I am trying to read a very big file (almost 1 GB) from S3. However the getobject fails to read the file with the error

WARNING: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.

I checked a few posts and found out that s3ObjectInputStream.abort() might solve the problem. But that still does not help.

    S3Object sourceS3Object = null;
    S3ObjectInputStream s3ObjectInputStream=null;
    InputStream reader = null;
    ObjectMetadata metadata = null;
    int retries = 10;

    while (retries > 0) {
        try {
            sourceS3Object = sourceS3Client.getObject(sourceS3Uri.getBucket(), sourceS3Uri.getKey());
            s3ObjectInputStream = sourceS3Object.getObjectContent();
            reader = new BufferedInputStream(s3ObjectInputStream);
        } catch (Exception readingException) {
            retries--;
            System.out.println(readingException);
            System.out.println(readingException.getStackTrace().toString());
            readingException.printStackTrace();

            if (s3ObjectInputStream != null) {
                s3ObjectInputStream.abort();
            }
            sourceS3Object = sourceS3Client.getObject(sourceS3Uri.getBucket(), sourceS3Uri.getKey());
            s3ObjectInputStream = sourceS3Object.getObjectContent();
        } finally {
            if (retries == 0) {
                System.out.println("Out of retries");
            }
        }
    }

The copy succeeds for smaller files but for larger files it gives me this warning

Answer 1

A bit belated but there is an obvious problem to fix.

You need to remove these two lines from your catch (Exception readingException) { block:

sourceS3Object = sourceS3Client.getObject(sourceS3Uri.getBucket(), sourceS3Uri.getKey());
s3ObjectInputStream = sourceS3Object.getObjectContent();

This is because when an exception occurs, you do this:

   ...
   s3ObjectInputStream = sourceS3Object.getObjectContent();

and then loop again. The first thing your loop does is the same:

   ...
   s3ObjectInputStream = sourceS3Object.getObjectContent();

so your program is getting the content stream two times, but only reading the content once. You need to remove these two lines from your Exception block:

Your loop also runs forever which I expect means you just stripped out some code for the example.

Answer 2

The warning mentions that, the file is closed without reading the whole file. In your code it is mentioned that,

 if (s3ObjectInputStream != null) {
                s3ObjectInputStream.abort();
            }

s3ObjectInputstream stores the file contents and it is being aborted without reading the whole file. So the code should be:

if (s3ObjectInputStream == null) {
                s3ObjectInputStream.abort();
            }