明确地说，如何让 s3fs 在 AWS 中打开 gz 文件？ 胶水/S3

Question

Using AWS glue and AWS s3 and s3fs, I've come up with the following (among other attempts). I even see the examples at https://s3fs.readthedocs.io/en/latest/ but they're not getting me there.

 key = 'https://s3.console.aws.amazon.com/s3/buckets/datalake-sbox-landing-55555-us-weast/cloudwatch_logs/prod-coarsair/dt%267D2019-09-11/144221-3837-462d-a2e6-ba300/coarsair-coarsair-144221-3837-462d-a2e6-ba300/000000.gz' s3 = s3fs.S3FileSystem() with s3.open(key, 'r') as f: args_gz_file = f

Then it should be as easy as typing "args_gz_file" now, right? Nope. Where am I going wrong?!

Answer 1

The samples show:

with fs.open('my-bucket/my-file.txt', 'rb') as f:

This suggests that the first parameter is BUCKET-NAME/KEY .

However, your code sample shows a URL ( https://s3.console.aws.amazon.com/s3/buckets/ ) instead of a bucket name.

Perhaps try:

key = 'datalake-sbox-landing-55555-us-weast/cloudwatch_logs/prod-coarsair/dt%267D2019-09-11/144221-3837-462d-a2e6-ba300/coarsair-coarsair-144221-3837-462d-a2e6-ba300/000000.gz'

Side-note: It is recommended to use the official AWS SDK or the AWS CLI to access Amazon S3, rather than using s3fs . Amazon S3 is an object storage service, not a filesystem.