AWS Storage gateway for caching millions of files in S3

Question

We have a use case where in we need to access almost millions of files from a Java application. Currently we are storing them in EBS volume. This is turning out to be expensive option(as we have reached upto 15TB now) so we are looking for S3 as the file storage. We are okay to bear the latency.

One option is to mount S3 using s3fs and access the files. But I was exploring the option of AWS Storage gateway if that can provide better caching and faster access. We have faced quite a few issues with s3fs so was looking for alternatives.

Answer 1

Avoid using s3fs if possible because it merely emulates a file system and is likely to run into problems with high utilization.

The best solution is for your application to access the files directly from Amazon via S3 API calls , rather than pretending that S3 is a filesystem. This works very nicely for large-scale applications and you would have no administration/maintenance overhead because your application communicates directly with S3. You should serious consider this option.

If you do really need to access the files via a filesystem, consider using AWS Storage Gateway – File Gateway , which can present S3 storage as an NFS share.