Prometheus Docker on AWS with EFS - No write access

Question

I'm running prometheus on ECS.

I'm mounting an EFS volume to my EC2 instance. When mounting the EFS I'm running chmod 777 on it. I'm attaching the EFS volume to the task-definition and then creating a mount point from the EFS volume the /prometheus container path.

When the container starts, it crashes with:

level=error ts=2020-08-10T16:04:39.961Z caller=query_logger.go:109 component=activeQueryTracker msg="Failed to create directory for logging active queries"

It's definitely permissions issue, since without mounting the volume it works fine. I also know that sometimes running chmod 777 won't suffice (for example running grafana the same way required to run chown 472:472 where 472 is grafana's user id), but I couldn't find what else to run.

Any ideas?

Answer 1

you can check if EFS file system policy has client root access enabled.

For troubleshooting, can check stopped tasks section.by clicking on any stopped task id you can see stopped reason.