如何使用AWS Data Pipeline将文件从一个S3存储桶/目录传输到另一个

Question

I would like to transfer a file (ie copy it to a target directory and delete it from the source directory) from one S3 bucket directory to another using AWS data pipeline.

I tried using the ShellCommandActivity and made a script that would move a file/files from one S3 bucket/directory to another. But the result was that it only copied it to the target S3 bucket/directory and did not remove the file in the S3 source directory.

Thanks in advance!

Answer 1

If you want to remove something from an S3 bucket you must explicitly delete it; there is no native concept of "move" other than copy and delete.

That said, last time I tried I found I could not delete from an S3 bucket resource in the data pipeline. I had to get around this by using s3cmd instead.