hadoop fs -ls s3://bucket 或 s3a://bucket 抛出“没有这样的文件或目录”错误

Question

In a newly created EMR cluster, using:

• hdfs dfs -ls s3://bucket
• hadoop fs -ls s3://bucket
• hadoop fs -ls s3a://
• etc.

...all return the error:

"ls: `s3://bucket': No such file or directory"

• EMR instance profile has full S3 access
• Nothing specified in core-site.xml
• aws s3 ls can correctly list all buckets

Why does this happen?

Answer 1

By default, hadoop fs -ls shows user home directory , which translates to /user/username .

When calling hadoop fs -ls s3://bucket , the S3 connector will try to find s3://bucket/user/hadoop (substitute tailing hadoop with your username) which may not exist and will lead to the error.

The error is not clear, but is different from ls ing a non-existant bucket. For that the error would be ls: Bucket bucket_name does not exist .

To avoid this:

• Append a / after the bucket name
• Add full path

To debug this:

• export HADOOP_ROOT_LOGGER=DEBUG,console

To turn off debug log

• export HADOOP_ROOT_LOGGER=WARN,console