Can't install cloudwatch agent by cloudformation on Amazon ECS-optimized AMI

Question

I am creating a cloudformation template, which creates some resources as EC2 instance, autoscaling group and launchConfiguration . By the userData property of the launchConfiguration resource, I tried to install the Cloudwatch agent as follows:

"UserData":{  "Fn::Base64" : {
            "Fn::Join" : ["", [
              "#!/bin/bash -xe\n",
              "yum -y install aws-cfn-bootstrap\n",
              "/opt/aws/bin/cfn-init -v",
              "         --stack ", { "Ref": "AWS::StackName" },
              "         --resource LaunchCongig",
              "         --region ", { "Ref" : "AWS::Region" },"\n",
              "yum -y install wget\n",
              "# Get the CloudWatch Logs agent\n",
              "wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py\n",
              "# Install the CloudWatch Logs agent\n",
              "python ./awslogs-agent-setup.py -n -r ", { "Ref" : "AWS::Region" }, " -c /etc/cwlogs.cfg || error_exit 'Failed to run CloudWatch Logs agent setup'\n",
              "service awslogs start"
            ]]}

After ssh into the instance, I checked the file /var/log/cloud-init-output.log to see if everything is fine, but here is what I got:

+ wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
--2017-02-17 14:36:10--  https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.226.59
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.226.59|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 47998 (47K) [text/x-python]
Saving to: ‘awslogs-agent-setup.py’

     0K .......... .......... .......... .......... ......    100%  196K=0.2s

2017-02-17 14:36:10 (196 KB/s) - ‘awslogs-agent-setup.py’ saved [47998/47998]

+ python ./awslogs-agent-setup.py -n -r eu-west-1 -c /etc/cwlogs.cfg

Step 1 of 5: Installing pip ...Traceback (most recent call last):
  File "./awslogs-agent-setup.py", line 1144, in <module>
    main()
  File "./awslogs-agent-setup.py", line 1140, in main
    setup.setup_artifacts()
  File "./awslogs-agent-setup.py", line 693, in setup_artifacts
    self.install_pip()
  File "./awslogs-agent-setup.py", line 600, in install_pip
    fail("Could not install pip. Please try again or see " + AGENT_SETUP_LOG_FILE + " for more details")
TypeError: fail() takes exactly 2 arguments (1 given)
+ error_exit 'Failed to run CloudWatch Logs agent setup'
/var/lib/cloud/instance/scripts/part-001: line 8: error_exit: command not found
Feb 17 14:36:12 cloud-init[2798]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [127]
Feb 17 14:36:12 cloud-init[2798]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Feb 17 14:36:12 cloud-init[2798]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 0.7.6 finished at Fri, 17 Feb 2017 14:36:12 +0000. Datasource DataSourceEc2.  Up 85.78 seconds

What is wrong with this script? Is there any other way to install the agent? Thank you.

EDIT:

I figured out that is because maybe the python-pip package didn't get installed so I added this to the userData :

  "yum -y install python-pip\n",

After that I played the template again and strangely I got the same Error.

I am usinh an Amazon ECS-optimized AMI

Answer 1

I solved the problem by installing the agent directly by yum awslogs :

      "UserData":{  "Fn::Base64" : {
        "Fn::Join" : ["", [
          "#!/bin/bash -xe\n",
          "yum -y install aws-cfn-bootstrap\n",
          "/opt/aws/bin/cfn-init -v",
          "         --stack ", { "Ref": "AWS::StackName" },
          "         --resource launchConfig",
          "         --region ", { "Ref" : "AWS::Region" },"\n",
          "yum -y install awslogs\n",
          "service awslogs start"
        ]]}

Here is the output from the log file:

Installed:
  awslogs.noarch 0:1.1.2-1.10.amzn1                                             

Dependency Installed:
  aws-cli.noarch 0:1.11.29-1.45.amzn1                                           
  aws-cli-plugin-cloudwatch-logs.noarch 0:1.3.3-1.15.amzn1                      
  freetype.x86_64 0:2.3.11-15.14.amzn1                                          
  libjpeg-turbo.x86_64 0:1.2.90-5.14.amzn1                                      
  mailcap.noarch 0:2.1.31-2.7.amzn1                                             
  python27-botocore.noarch 0:1.4.86-1.62.amzn1                                  
  python27-colorama.noarch 0:0.2.5-1.7.amzn1                                    
  python27-dateutil.noarch 0:2.1-1.3.amzn1                                      
  python27-docutils.noarch 0:0.11-1.15.amzn1                                    
  python27-futures.noarch 0:3.0.3-1.3.amzn1                                     
  python27-imaging.x86_64 0:1.1.6-19.9.amzn1                                    
  python27-jmespath.noarch 0:0.9.0-1.11.amzn1                                   
  python27-ply.noarch 0:3.4-3.12.amzn1                                          
  python27-pyasn1.noarch 0:0.1.7-2.9.amzn1                                      
  python27-rsa.noarch 0:3.4.1-1.8.amzn1                                         

Complete!
+ service awslogs start
Starting awslogs: [  OK  ]
Cloud-init v. 0.7.6 finished at Fri, 17 Feb 2017 15:33:42 +0000. Datasource DataSourceEc2.  Up 83.47 seconds

Everything works fine this way. Hope that will help someone someday!

Answer 2

For ECS specifically, see Using CloudWatch Logs with Container Instances in the EC2 Container Service documentation for details on configuring CloudWatch Logs. The documentation recommends using yum install -y awslogs instead of the Python install script.

The documentation provides a complete sample in the Configuring CloudWatch Logs at Launch with User Data section.

In your case, since you're already managing your config files using cfn-init and CloudFormation::Init metadata in CloudFormation, you don't need any complex parsing of config files in your User-Data script, but you can still use the script as a reference. One thing worth adding to your User-Data script is running chkconfig awslogs on to make sure the service continues running on the instance after a reboot.