访问 AWS ParallelCluster 上的 s3 存储桶

Question

I have a requirement of accessing S3 bucket on the AWS ParallelCluster nodes. I did explore the s3_read_write_resource option in the ParallelCluster documentation. But, it is not clear as to how we can access the bucket. For example, will it be mounted on the nodes, or will the users be able to access it by default. I did test the latter by trying to access a bucket I declared using the s3_read_write_resource option in the config file, but was not able to access it ( aws s3 ls s3://<name-of-the-bucket> ).

I did go through this github issue talking about mounting S3 bucket using s3fs. In my experience it is very slow to access the objects using s3fs.

So, my question is,

How can we access the S3 bucket when using s3_read_write_resource option in AWS ParallelCluster config file

Answer 1

These parameters are used in ParallelCluster to include S3 permissions on the instance role that is created for cluster instances. They're mapped into Cloudformation template parameters S3ReadResource and S3ReadWriteResource. And later used in the Cloudformation template. For example, here and here . There's no special way for accessing S3 objects.

To access S3 on one cluster instance, we need to use the aws cli or any SDK. Credentials will be automatically obtained from the instance role using instance metadata service.

Please note that ParallelCluster doesn't grant permissions to list S3 objects.

Retrieving existing objects from S3 bucket defined in s3_read_resource, as well as retrieving and writing objects to S3 bucket defined in s3_read_write_resource should work.

However, "aws s3 ls" or "aws s3 ls s3://name-of-the-bucket" need additional permissions. See https://aws.amazon.com/premiumsupport/knowledge-center/s3-access-denied-listobjects-sync/ .

I wouldn't use s3fs, as it's not AWS supported, it's been reported to be slow (as you've already noticed), and other reasons .

You might want to check the FSx section . It can create an attach an FSx for Lustre filesystem. It can import/export files to/from S3 natively. We just need to set import_path and export_path on this section.