How to handle running out of direct memory when downloading large files with Vert.x & Netty?

Question

I have a Vert.x web service that will occasionally download large ZIP files from AWS S3. After downloading, the archive is unzipped and individual files are re-uploaded to AWS S3. The web service is hosted as a t2.large (8GB memory) instance in AWS Elastic Beanstalk. The Java application is currently configured with between 2-4GB of heap space, and the ZIP files will be at most 10GB in size (but most will be closer to 2-4GB at most).

When the application tries to download ZIP files >2GB in size, either the initial download of the ZIP file or the re-upload of individual files always fails with a stack trace similar to the following:

Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1895825439, max: 1908932608)

After doing some research, it appears that Vert.x uses Netty to speed up Network I/O, which in turn utilizes direct memory to improve download performance. It appears that the direct memory isn't being freed sufficiently quickly, which leads to out-of-memory exceptions like the above.

The simplest solution would just be to increase the instance size to 16GB t2.xlarge and allocate more direct memory at runtime (eg. -XX:MaxDirectMemorySize ), but I'd like to explore other solutions first. Is there a way to programmatically force Netty to free direct memory after it's no longer in use? Is there additional Vert.x configuration I can add that might alleviate this problem?

Answer 1

Please check this

github.com/aws/aws-sdk-java-v2/issues/1301

we have identified an issue within the SDK where it could cause excessive buffer usage and eventually OOM when using s3 async client to download a large object to a file. The fix #1335 is available in 2.7.4. Could you try with the latest version? Feel free to re-open if you continue to see the issue. " – AWS PS 21 hours ago Delete