AWS cli throws error when copying large files

Question

I'm trying to copy objects from an s3 bucket to another using aws cli tool.

It works OK for small objects, but on large file buckets, as soon as the copy starts, I get one of the following errors:

copy failed: s3://bucket/file.ogv to s3://bucket-tmp/file.ogv ('Connection aborted.', OSError(0, 'Error'))

or

copy failed: s3://bucket/file.ogv to s3://bucket-tmp/file.ogv An error occurred (NoSuchKey) when calling the UploadPartCopy operation: Unknown

if I include the --no-guess-mime-type I get

fatal error: ('Connection aborted.', OSError(0, 'Error'))

I tryied --debug, but I really didn't understand much of the debug output but I could see OSError(0, 'Error') again in the log.

Anyone has seen anything like this? in another answer ( this one ), people told about another tool s3cmd, but I couldn't make it work.

I'm trying to access ceph on a corporate server with path-style urls and https endpoint.

My command:

aws --endpoint-url https://myendpoint.url s3 cp s3://mybucket s3://mybucket-tmp --recursive

Also when I tried to configure s3cmd I get an ungly python debug output with OSError: [Errno 0] Error in the middle.

Answer 1

I discovered that if I use s3api command instead of s3 command it works. Format of working command:

aws --endpoint-url <my-endpoint-url> s3api copy-object --copy-source my-source-bucket/whatever/path/file.txt --key whatever/path/file.txt --bucket my-destination-bucket

It only copys one file at once. You can grab a list of objects in the bucket using s3 command ls or s3api command list-objects