How to rename “PREFIX” in Amazon s3 via aws cli?

Question

I understand that in Amazon s3, there are only buckets and objects . There are no such things as folders and all files sit in the same bucket, and that objects that appear to be in a folder only have a prefix in front of their file name.

Now my question is... is there a way to change the PREFIX using AWS CLI?

Answer 1

In awscli2 I was able to accomplish this with the following command:

aws s3 mv --recursive s3://bucket/prefix1/ s3://bucket/prefix2/

I ran a quick test (dryrun) prior to doing this by checking the output of:

aws s3 --dryrun mv --recursive s3://bucket/prefix1/ s3://bucket/prefix2/

Answer 2

Yes, but not easily. Keeping in mind that a "prefix" is just part of a filename, changing a prefix will require that you rename every file that has that prefix in it — for instance, if your bucket contained the following files, and you wanted to rename /oldprefix/ to /newprefix/ :

/oldprefix/file1.txt
/oldprefix/file2.txt
/oldprefix/folder/anotherfile.txt

You would need to perform three* operations to accomplish this change of prefix: one for each object. There is no way to rename them all in a single operation. In general, you should try to avoid situations where you need to do this kind of renaming on a large scale, as it can be a rather lengthy process.

*: If you want to get picky about it, there are actually not three but six operations required, since renaming an object is accomplished in two steps by first copying it to the new name, then deleting the original. Many S3 clients and libraries will handle this detail for you, though.

Answer 3

[edit] I realized that my answer didn't fully answer the OP, and as several people pointed out, could use some more information.

In general, the AWS CLI for S3 will look for objects under a specified bucket AND prefix. If the prefix is included, it ignores that portion of an object's name when matching objects.

For example, say you have a number of images two folders deep in a bucket backup . Your object name may look like this photos/graduation/image1.jpg .

You can copy the inner folder ( graduation in this case), as well as all files in that folder, to the top level with the following command:

aws s3 cp s3://backup/photos s3://backup --recursive

This matches all files in the bucket backup with the prefix photos . It will copy those files to the backup bucket, and name them whatever is left after stripping of the prefix - so in this case, they will be named graduation/image1.jpg .

Note that the original files will still be in the original folder. To truly simulate a rename of the prefix, you need to remove the old files.

aws rm s3://backup/photos/graduation --recursive 

Once you are done, you will have a graduation folder of images under the root level of your bucket, and none under the photos folder. In other words, you have renamed the prefix from photos/graduation to graduation .

Unfortunately it does not appear that you can use the mv command to do this within the same bucket - it fails with an error saying "Cannot mv a file onto itself".

Answer 4

Ran into the same problem. Below is how I solved it with the aws cli from Linux bash.

# check current contents
aws s3 ls --human-readable s3://mybucket/

Output:

PRE AK/
PRE AR/
PRE DC/
PRE DE/
PRE HI/
PRE OH/
PRE TN/
PRE VT/

Construct array to iterate over then iterate and execute aws s3 mv on each existing prefix and move into a new prefix, reusing the existing prefix as needed.

prefixArr=(AK AR DC DE HI OH TN VT)

for s3p in ${prefixArr2[@]}; do
     echo moving s3://mybucket/$s3p
     aws s3 mv s3://mybucket/$s3p s3://mybucket/state=$s3p.parquet --recursive
done

Output:

moving s3://mybucket/AK
move: s3://mybucket/AK/_SUCCESS to s3://mybucket/state=AK.parquet/_SUCCESS
move: s3://mybucket/AK/part-00002.snappy.parquet to s3://mybucket/state=AK.parquet/part-00002.snappy.parquet
move: s3://mybucket/AK/part-00003.snappy.parquet to s3://mybucket/state=AK.parquet/part-00003.snappy.parquet
...

Check results:

aws s3 ls --human-readable s3://mybucket/

Output:

PRE state=AK.parquet/
PRE state=AR.parquet/
PRE state=DC.parquet/
PRE state=DE.parquet/
PRE state=HI.parquet/
PRE state=OH.parquet/
PRE state=TN.parquet/
PRE state=VT.parquet/