[英]Cannot write files on mounted volume aws efs
I'm building a scalable and distributed architecture for workflows within our company by leveraging the use of Apache Airflow. I'm using ECS Fargate.我正在利用 Apache Airflow 的使用为我们公司内的工作流构建一个可扩展的分布式架构。我正在使用 ECS Fargate。
Since this is a distributed architecture, a non-centralized file system is required in order to have a consistent-shared view across all machines (ie, the webserver needs to access the DAGs files, scheduler, and workers also).由于这是一个分布式架构,因此需要一个非集中式文件系统,以便在所有机器上具有一致的共享视图(即,网络服务器需要访问 DAG 文件、调度程序和工作程序)。
For such purpose, I'm using AWS EFS, I can successfully mount the file system into an EC2 instance, but I'm unable to write or create a file within it.为此,我正在使用 AWS EFS,我可以成功地将文件系统安装到 EC2 实例中,但我无法在其中写入或创建文件。
This is the policy attached to the fs:这是附加到 fs 的策略:
{
"Version": "2012-10-17",
"Id": "ExamplePolicy01",
"Statement": [
{
"Sid": "ExampleSatement01",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"elasticfilesystem:ClientMount",
"elasticfilesystem:ClientWrite"
],
"Resource": "arn:aws:elasticfilesystem:eu-west-1:XXXXXXXXXX:file-system/fs-aaaaaaa"
}
]
}
and as the doc says:正如文档所说:
elasticfilesystem:ClientWrite ------> Provides an NFS client with write permissions on a file system.
elasticfilesystem:ClientWrite ------> 为 NFS 客户端提供对文件系统的写入权限。
I'm mounting by executing the following command:我通过执行以下命令进行安装:
sudo mount -t efs -o tls,accesspoint=fsap-oooooooooooooooo fs-aaaaaaa:/ efs
and indeed, I can see the content, but got Permission denied
every time I try to write.事实上,我可以看到内容,但每次尝试写入时都
Permission denied
。
[ec2-user@ip-XXXX ~]$ sudo mount -t efs -o tls,accesspoint=fsap-oooooooooooooooo fs-aaaaaaa:/ efs
[ec2-user@ip-XXXX ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 616K 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/xvda1 8.0G 2.7G 5.4G 33% /
tmpfs 395M 0 395M 0% /run/user/1000
tmpfs 395M 0 395M 0% /run/user/0
127.0.0.1:/ 8.0E 0 8.0E 0% /home/ec2-user/efs
[ec2-user@ip-XXXX ~]$
[ec2-user@ip-XXXX ~]$ cd efs/
[ec2-user@ip-XXXX efs]$ touch a
touch: cannot touch ‘a’: Permission denied
[ec2-user@ip-XXXX efs]$
[ec2-user@ip-XXXX efs]$ sudo touch a
touch: cannot touch ‘a’: Permission denied
[ec2-user@ip-XXXX efs]$ sudo su -
Last login: Wed Aug 19 21:29:39 UTC 2020 on pts/0
[root@ip-XXXX ~]#
[root@ip-XXXX ~]# cd /home/ec2-user/efs/
[root@ip-XXXX efs]#
[root@ip-XXXX efs]# touch d
touch: cannot touch ‘d’: Permission denied
Any help would be much appreciated.任何帮助将非常感激。 Thank you.
谢谢你。
I have successfully achieved the goal, and this answer is intended to share my experience just in case someone bumps into the same use case.我已经成功实现了目标,这个答案旨在分享我的经验,以防有人遇到相同的用例。
If you are using ECS Fargate, there is an option to mount volumes in the version with platform 1.4.0.如果您使用的是 ECS Fargate,则可以选择在平台 1.4.0 的版本中装载卷。 So, within
container_definitions
there is an option called mountPoints
:因此,在
container_definitions
有一个名为mountPoints
的选项:
...
"entryPoint": [],
"command": [],
"portMappings": [
{
"hostPort": ${host_port},
"protocol": "tcp",
"containerPort": ${container_port}
}
],
"mountPoints": [
{
"sourceVolume": "dags",
"containerPath": "/usr/local/airflow/dags",
"readOnly": false
}
],
"cpu": ${cpu},
"environment": [
{
"name": "POSTGRES_HOST",
"value": "${postgres_host}"
},
{
"name": "AIRFLOW_COMPONENT",
"value": "webserver"
},
{
"name": "REDIS_HOST",
"value": "${redis_host}"
}
]
...
that will mount the EFS volume.这将挂载 EFS 卷。 Remember to specify the
volume
configuration.请记住指定
volume
配置。 Here's the terraform:这是 terraform:
resource "aws_ecs_task_definition" "task_definition_airflow_webserver" {
family = "airflow-webserver"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
execution_role_arn = "${aws_iam_role.ecs_service_role_airflow_webserver.arn}"
container_definitions = "${data.template_file.airflow_webserver.rendered}"
cpu = 1024
memory = 2048
volume {
name = "dags"
efs_volume_configuration {
file_system_id = "${aws_efs_file_system.airflow_dags_efs.id}"
root_directory = "/"
transit_encryption = "ENABLED"
}
}
}
Hope this helps.希望这可以帮助。 Let me know if there's something ambiguous or unclear.
让我知道是否有任何模棱两可或不清楚的地方。
I also encountered this problem.我也遇到了这个问题。 The solution that worked for me was two steps
对我有用的解决方案是两个步骤
Add elasticfilesystem:ClientRootAccess
to the action statement of the policy.将
elasticfilesystem:ClientRootAccess
添加到策略的操作语句中。 For example,例如,
"Action": [ "elasticfilesystem:ClientRootAccess", "elasticfilesystem:ClientMount", "elasticfilesystem:ClientWrite" ],
Then, after mounting the EFS, change it's permissions to allow writing.然后,在安装 EFS 后,更改它的权限以允许写入。 For example if I mounted to
/efs/data
, then例如,如果我安装到
/efs/data
,那么
cd /efs/data sudo chmod 775 /efs/data
And now it works.现在它起作用了。
If you want to avoid step 2, you can go into the AWS "Access Point" settings and update the "POSIX User" and "Root directory creation permissions" to be your user/group IDs.如果您想避免第 2 步,您可以 go 进入 AWS“访问点”设置并将“POSIX 用户”和“根目录创建权限”更新为您的用户/组 ID。 For example, in ubuntu the default user usually has a POSIX ID of "1000" and a group ID of "1000".
例如,在 ubuntu 中,默认用户通常具有“1000”的 POSIX ID 和“1000”的组 ID。
To find out your user ID,要找出您的用户 ID,
id -u <username>
To find out your group ID,要找出您的组 ID,
id -g <username>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.