AWS CodePipeline Fails attaching EFS to Multicontainer Docker Elastic Beanstalk

Question

I use AWS CodePipeline to deploy a multicontainer docker app on Elastic Beanstalk. The app requires attaching a file system from EFS as a mounted volume. In order to attach and have access to the volume, it appears I need to restart docker. This is explained in this question: Using AWS EFS with Docker . While I can successfully attach the file system based on those instructions, my deployment sometimes “fails” with the following error:

Action execution failed Deployment completed, but with errors: ECS task stopped due to: Task failed to start.  Failed to start ECS task: arn:aws… is STOPPED.

In these cases the app successfully deploys, and is fully functional, but I receive a failure message because docker had not restarted yet during that check.

When this occurs I see the following in my Beanstalk logs:

-------------------------------------
/var/log/ecs/ecs-init.log
-------------------------------------
2020-04-27T15:06:41Z [INFO] pre-start
2020-04-27T15:06:44Z [INFO] start
2020-04-27T15:06:44Z [INFO] No existing agent container to remove.
2020-04-27T15:06:44Z [INFO] Starting Amazon Elastic Container Service Agent
2020-04-27T15:07:20Z [INFO] Agent exited with code 0
2020-04-27T15:07:20Z [INFO] Error connecting to docker, backing off for 1.14777941s, error: Get http://unix.sock/v1.25/version: dial unix /var/run/docker.sock: connect: no such file or directory
2020-04-27T15:07:21Z [INFO] Error connecting to docker, backing off for 2.282153551s, error: Get http://unix.sock/v1.25/version: dial unix /var/run/docker.sock: connect: no such file or directory
2020-04-27T15:07:23Z [INFO] post-stop
2020-04-27T15:07:23Z [INFO] Cleaning up the credentials endpoint setup for Amazon Elastic Container Service Agent
2020-04-27T15:07:23Z [INFO] pre-start
2020-04-27T15:07:23Z [INFO] start
2020-04-27T15:07:23Z [INFO] Container name: /ecs-agent
2020-04-27T15:07:23Z [INFO] Removing existing agent container ID:
2020-04-27T15:07:23Z [INFO] Starting Amazon Elastic Container Service Agent

Is there a way to prevent these false failures?

Answer 1

The problem seems that docker daemon is not running at the time when ecs-agent is started. Elasticbeanstalk has its own way of managing docker daemon and ecs-agent start up unless it is intervened manually/automatically.

Can you please confirm if you restarting docker/ecs-agent via any EB extensions, as it may hamper the EB normal bootstrap process?

If you are using EB extensions to start/restart docker, please do it in 'post-deploy' phase using 'hooks' that will not affect your bootstrap process. Here are the common hooks you can use [1]. There are some examples given on this link [2] as well.

References:

[1] https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/custom-platform-hooks.html

[2] https://github.com/equivalent/scrapbook2/blob/master/archive/blogs/2016-08-22-aws-elasticbeanstalk-hooks.md