[英]EC2 instance running S3 Sync command terminates before data transfer is complete
I have an EC2 instance running Linux. This instance is used to run aws s3
commands.我有一个运行 Linux 的 EC2 实例。此实例用于运行
aws s3
命令。
I want to sync the last 6 months worth of data from source
to target
S3 buckets.我想将过去 6 个月的数据从
source
同步到target
S3 存储桶。 I am using credentials with the necessary permissions to do this.我正在使用具有必要权限的凭据来执行此操作。
Initially I just ran the command:最初我只是运行命令:
aws s3 sync "s3://source" "s3://target" --query "Contents[?LastModified>='2022-08-11' && LastModified<='2023-01-11']"
However, after maybe 10 mins this command stops running, and only a fraction of the data is synced.但是,大约 10 分钟后,此命令停止运行,并且只有一小部分数据被同步。
I thought this was because my SSM session was terminating, and with it the command stopped executing.我认为这是因为我的 SSM session 正在终止,并且命令停止执行。
To combat this, I used the following command to try and ensure that this command would continue to execute even after my SSM terminal session was closed:为了解决这个问题,我使用了以下命令来尝试确保即使在我的 SSM 终端 session 关闭后该命令也会继续执行:
nohup aws s3 sync "s3://source" "s3://target" --query "Contents[?LastModified>='2022-08-11' && LastModified<='2023-01-11']" --exclude "*.log" --exclude "*.bak" &
Checking the status of the EC2 instance, the command appears to run for about 20 mins, before clearly stopping for some reason.检查 EC2 实例的状态,该命令似乎运行了大约 20 分钟,然后由于某种原因明显停止。
The --query
parameter controls what information is displayed in the response from an API call. --query
参数控制在 API 调用的响应中显示的信息。
It does not control which files are copied in an aws s3 sync
command.它不控制在
aws s3 sync
命令中复制哪些文件。 The documentation for aws s3 sync
defines the --query
parameter as: "A JMESPath query to use in filtering the response data." aws s3 sync
的文档将--query
参数定义为: “用于过滤响应数据的 JMESPath 查询。”
Your aws s3 sync
command will be synchronizing ALL files unless you use Exclude and Include Filters .您的
aws s3 sync
命令将同步所有文件,除非您使用Exclude 和 Include Filters 。 These filters operate on the name of the object. It is not possible to limit the sync
command by supplying date ranges.这些过滤器对 object 的名称进行操作。无法通过提供日期范围来限制
sync
命令。
I cannot comment on why the command would stop running before it is complete.我无法评论为什么命令会在完成之前停止运行。 I suggest you redirect output to a log file and then review the log file for any clues.
我建议您将 output 重定向到日志文件,然后查看日志文件以查找任何线索。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.