如何使用CloudFormation模板从S3获取用于引导应用程序的文件，而无需启动具有角色的EC2实例？

Question

I need to download a file from S3 bucket which is a private location. I have created an IAM Role using AWS Console - hence an have an IAM Profile which I assign to instances launched. I can get the file from secure location by providing an Authentication Object in LaunchConfiguration. This only works if the instance is launched with an IamInstanceProfile .

   "LaunchConfiguration":{
        "Type":"AWS::AutoScaling::LaunchConfiguration",
        "Metadata":{
            "Comment":"Configure the Presentation Tier",
             "AWS::CloudFormation::Authentication": {
                "S3Access" : {
                    "type" : "S3",
                    "buckets":["demo-upload"],
                    "roleName": "MYS3ROLE"
                }
            },
            "AWS::CloudFormation::Init":{
                "config":{
                    "packages":{
                        "yum":{
                            "tomcat6": [],
                            "java-1.6.0-openjdk-devel" : [],
                            "tomcat6-webapps" :[],
                            "tomcat6-admin-webapps" :[],
                            "tomcat6-docs-webapp" :[],
                            "curl":[]
                        }
                    },
                    "files":{
                        "/var/lib/tomcat6/webapps/demo.war" : { 
                            "source" : {"Ref":"WarLoc"},
                            "owner" : "tomcat",
                            "group" : "tomcat",
                            "authentication":"S3Access"
                        },
                        "/root/demo.war" : { 
                            "source" : {"Ref":"WarLoc"},
                            "owner" : "root",
                            "group" : "root",
                            "authentication":"S3Access"
                        },
                        "/etc/cfn/cfn-hup.conf" : {
                            "content" : { "Fn::Join" : ["", [
                            "[main]\n",
                            "stack=", { "Ref" : "AWS::StackId" }, "\n",
                            "region=", { "Ref" : "AWS::Region" }, "\n",
                            "interval=1"
                            ]]},
                        "mode"    : "000400",
                        "owner"   : "root",
                        "group"   : "root"
                        },

                      "/etc/cfn/hooks.d/cfn-auto-reloader.conf" : {
                        "content": { "Fn::Join" : ["", [
                          "[cfn-auto-reloader-hook]\n",
                          "triggers=post.update\n",
                          "path=Resources.LaunchConfiguration.Metadata.AWS::CloudFormation::Init\n",
                          "action=/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" },
                          "         -r LaunchConfiguration ",
                          "         --region     ", { "Ref" : "AWS::Region" }, "\n",
                          "runas=root\n"
                        ]]}
                      }
                    },
                    "services" : {
                        "sysvinit" : {
                            "tomcat6":{
                                "files":["/var/lib/tomcat6/webapps/demo.war"],
                                "enabled": "true",
                                "ensureRunning":"true"
                            },
                            "cfn-hup":{
                                "files":["/etc/cfn/cfn-hup.conf","/etc/cfn/hooks.d/cfn-auto-reloader.conf"],
                                "enabled":"true",
                                "ensureRunning":"true"
                            }
                        }
                    }
                }
            }
        },
        "Properties":{
            "ImageId"        : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
            "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
            "InstanceType"   : { "Ref" : "InstanceType" },
            "InstanceMonitoring": "false",
            "KeyName"        : { "Ref" : "KeyName" },
            "AssociatePublicIpAddress" : "true",
            "IamInstanceProfile":"arn:aws:iam::MY_ACCOUNT_ID:instance-profile/MYS3ROLE",
            "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
              "#!/bin/bash -v\n",
              "yum update -y\n",
              "# Install Presentation Software\n",
              "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r LaunchConfiguration ",
              "    --region ", { "Ref" : "AWS::Region" }, "\n",
              "service tomcat6 stop\n",
              "rm -rf /var/lib/tomcat6/webapps/demo\n",
              "sleep 3\n",
              "service tomcat6 start\n",
              "sleep 3\n",
              "service tomcat6 restart\n",
              "/opt/aws/bin/cfn-hup\n",
              "/opt/aws/bin/cfn-signal -e $? -r \"Setup complete\" '", { "Ref" : "WaitHandle" }, "'\n"
              ]]}}
        }
    },

Can I download the files from S3 which is not public without having to launch the machine with an IamInstanceProfile . ?

If I delete the the "IamInstanceProfile":"arn:aws:iam::MY_ACCOUNT_ID:instance-profile/s3access", then in /var/log/cfn-init.log I find the following lines, which I don't think would be because of Authentication Faiures

ToolError: Failed to retrieve https://s3-us-west-2.amazonaws.com/demo-upload/0.0.1/demo.war: [Errno 404] HTTP Error 404 : <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
 <title>404 - Not Found</title>
</head>
<body>
   <h1>404 - Not Found</h1>
 </body>
</html>

When I launch the Instance with IamProfile everything runs fine. So, is there a way to get the files - without scripts and storing credentials on machine - but not have instance launched with a role ?

Answer 1

IAM Instance profile is the cleanest way to get a private file from S3 to EC2.

Other options are (apart from keeping key in code - which is not good!):

• Instance Metadata and User Data - Amazon Elastic Compute Cloud : http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
• Use a deployment service like Elastic Beanstalk or OpsWorks . You can pass anything to the EC2s started this way. In fact this is an easy way to manage clusters.

Answer 2

I don't know of a way to access S3 from an EC2 instance without credentials, but I do have an idea for you.

1. You can create your IAM Role inside your template, but without a policy (or an empty policy if semantics make you specify one).

2. Create an instance with this Role (You may need to make a WaitCondition).

3. After the instance has been created (you will need a WaitCondition), create an IAM Role Policy that has the permissions you need and which restricts access to the Instance's IP address. Have it attach to the Role you created.

Your instance should have access to the resources it needs before it fully boots.