如何将环境变量添加到CodeBuild buildspec.yml？

Question

I'm trying to use a CloudFormation template to define CodeBuild and CodePipeline to automate the deployment of a static website hosted in an S3 bucket. To give credit where credit's due, I'm largely following the template from https://dzone.com/articles/continuous-delivery-to-s3-via-codepipeline-and-cod .

The problem I can't resolve is that after I add an environment variable for the Hugo version I'd like to use to create the static site files, I get an error from the AWS console that reads: "Template validation error: Template format error: Unresolved resource dependencies [HUGO_VERSION] in the Resources block of the template."

Why isn't it accepting the HUGO_VERSION environment variable that I define under environment_variables? This is version 0.1 of the format, so it's a little different than the current 0.2, but I've read the following link: https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax

The thing that really confuses me is that if I remove the lines with ${HUGO_VERSION}, the template is accepted just fine - and then inspection of the CloudWatch logs after building shows (because of the printenv command) HUGO_VERSION=0.49! What gives?

Originally, the template looks like this.

---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
  SiteBucketName:
    Type: String
    Description: Name of bucket to create to host the website
  GitHubUser:
    Type: String
    Description: GitHub User
    Default: "stelligent"
  GitHubRepo:
    Type: String
    Description: GitHub Repo to pull from. Only the Name. not the URL
    Default: "devops-essentials"
  GitHubBranch:
    Type: String
    Description: GitHub Branch
    Default: "master"
  GitHubToken:
    NoEcho: true
    Type: String
    Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
  BuildType:
    Type: String
    Default: "LINUX_CONTAINER"
    Description: The build container type to use for building the app
  BuildComputeType:
    Type: String
    Default: "BUILD_GENERAL1_SMALL"
    Description: The build compute type to use for building the app
  BuildImage:
    Type: String
    Default: "aws/codebuild/ubuntu-base:14.04"
    Description: The build image to use for building the app
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: "Site Configuration"
        Parameters:
          - SiteBucketName
      - Label:
          default: "GitHub Configuration"
        Parameters:
          - GitHubToken
          - GitHubUser
          - GitHubRepo
          - GitHubBranch
      - Label:
          default: "Build Configuration"
        Parameters:
          - BuildType
          - BuildComputeType
          - BuildImage
    ParameterLabels:
      SiteBucketName:
        default: Name of S3 Bucket to create for website hosting
      GitHubToken:
        default: GitHub OAuth2 Token
      GitHubUser: 
        default: GitHub User/Org Name
      GitHubRepo: 
        default: GitHub Repository Name
      GitHubBranch: 
        default: GitHub Branch Name
      BuildType: 
        default: CodeBuild type
      BuildComputeType: 
        default: CodeBuild instance type
      BuildImage: 
        default: CodeBuild image
Resources:
  CodeBuildRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - codebuild.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: codebuild-service
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action: "*"
            Resource: "*"
          Version: '2012-10-17'
  CodePipelineRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - codepipeline.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: codepipeline-service
        PolicyDocument:
          Statement:
          - Action:
            - codebuild:*
            Resource: "*"
            Effect: Allow
          - Action:
            - s3:GetObject
            - s3:GetObjectVersion
            - s3:GetBucketVersioning
            Resource: "*"
            Effect: Allow
          - Action:
            - s3:PutObject
            Resource:
            - arn:aws:s3:::codepipeline*
            Effect: Allow
          - Action:
            - s3:*
            - cloudformation:*
            - iam:PassRole
            Resource: "*"
            Effect: Allow
          Version: '2012-10-17'
  SiteBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
    Properties:
      AccessControl: PublicRead
      BucketName: !Ref SiteBucketName
      WebsiteConfiguration:
        IndexDocument: index.html
  PipelineBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
  CodeBuildDeploySite:
    Type: AWS::CodeBuild::Project
    DependsOn: CodeBuildRole
    Properties:
      Name: !Sub ${AWS::StackName}-DeploySite
      Description: Deploy site to S3
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        Type: !Ref BuildType
        ComputeType: !Ref BuildComputeType
        Image: !Sub ${BuildImage}
      Source:
        Type: CODEPIPELINE
        BuildSpec: !Sub |
          version: 0.1
          phases:
            post_build:
              commands:
                - aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples 
                - aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/ 
          artifacts:
            type: zip
            files:
              - ./html/index.html
      TimeoutInMinutes: 10
  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      RoleArn: !GetAtt CodePipelineRole.Arn
      Stages:
      - Name: Source
        Actions:
        - InputArtifacts: []
          Name: Source
          ActionTypeId:
            Category: Source
            Owner: ThirdParty
            Version: '1'
            Provider: GitHub
          OutputArtifacts:
          - Name: SourceArtifacts
          Configuration:
            Owner: !Ref GitHubUser
            Repo: !Ref GitHubRepo
            Branch: !Ref GitHubBranch
            OAuthToken: !Ref GitHubToken
          RunOrder: 1
      - Name: Deploy
        Actions:
        - Name: Artifact
          ActionTypeId:
            Category: Build
            Owner: AWS
            Version: '1'
            Provider: CodeBuild
          InputArtifacts:
          - Name: SourceArtifacts
          OutputArtifacts:
          - Name: DeploymentArtifacts
          Configuration:
            ProjectName: !Ref CodeBuildDeploySite
          RunOrder: 1
      ArtifactStore:
        Type: S3
        Location: !Ref PipelineBucket
Outputs:
  PipelineUrl:
    Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
    Description: CodePipeline URL
  SiteUrl:
    Value: !GetAtt [SiteBucket, WebsiteURL]
    Description: S3 Website URL

Now after I try to add an environment variable to use Hugo in the pipeline, the template looks like this.

---
AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
  SiteBucketName:
    Type: String
    Description: Name of bucket to create to host the website
  GitHubUser:
    Type: String
    Description: GitHub User
    Default: "stelligent"
  GitHubRepo:
    Type: String
    Description: GitHub Repo to pull from. Only the Name. not the URL
    Default: "devops-essentials"
  GitHubBranch:
    Type: String
    Description: GitHub Branch
    Default: "master"
  GitHubToken:
    NoEcho: true
    Type: String
    Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
  BuildType:
    Type: String
    Default: "LINUX_CONTAINER"
    Description: The build container type to use for building the app
  BuildComputeType:
    Type: String
    Default: "BUILD_GENERAL1_SMALL"
    Description: The build compute type to use for building the app
  BuildImage:
    Type: String
    Default: "aws/codebuild/ubuntu-base:14.04"
    Description: The build image to use for building the app
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: "Site Configuration"
        Parameters:
          - SiteBucketName
      - Label:
          default: "GitHub Configuration"
        Parameters:
          - GitHubToken
          - GitHubUser
          - GitHubRepo
          - GitHubBranch
      - Label:
          default: "Build Configuration"
        Parameters:
          - BuildType
          - BuildComputeType
          - BuildImage
    ParameterLabels:
      SiteBucketName:
        default: Name of S3 Bucket to create for website hosting
      GitHubToken:
        default: GitHub OAuth2 Token
      GitHubUser: 
        default: GitHub User/Org Name
      GitHubRepo: 
        default: GitHub Repository Name
      GitHubBranch: 
        default: GitHub Branch Name
      BuildType: 
        default: CodeBuild type
      BuildComputeType: 
        default: CodeBuild instance type
      BuildImage: 
        default: CodeBuild image
Resources:
  CodeBuildRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - codebuild.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: codebuild-service
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action: "*"
            Resource: "*"
          Version: '2012-10-17'
  CodePipelineRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - codepipeline.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: codepipeline-service
        PolicyDocument:
          Statement:
          - Action:
            - codebuild:*
            Resource: "*"
            Effect: Allow
          - Action:
            - s3:GetObject
            - s3:GetObjectVersion
            - s3:GetBucketVersioning
            Resource: "*"
            Effect: Allow
          - Action:
            - s3:PutObject
            Resource:
            - arn:aws:s3:::codepipeline*
            Effect: Allow
          - Action:
            - s3:*
            - cloudformation:*
            - iam:PassRole
            Resource: "*"
            Effect: Allow
          Version: '2012-10-17'
  SiteBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
    Properties:
      AccessControl: PublicRead
      BucketName: !Ref SiteBucketName
      WebsiteConfiguration:
        IndexDocument: index.html
  PipelineBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
  CodeBuildDeploySite:
    Type: AWS::CodeBuild::Project
    DependsOn: CodeBuildRole
    Properties:
      Name: !Sub ${AWS::StackName}-DeploySite
      Description: Deploy site to S3
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        Type: !Ref BuildType
        ComputeType: !Ref BuildComputeType
        Image: !Sub ${BuildImage}
      Source:
        Type: CODEPIPELINE
        BuildSpec: !Sub |
          version: 0.1
          environment_variables:
            plaintext:
              AWS_DEFAULT_REGION: "US-WEST-2"
              HUGO_VERSION: "0.49"
          phases:
            install:
              commands:
                - printenv
                - echo "Install step..."
                - curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
                - tar xf /tmp/hugo.tar.gz -C /tmp
                - mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
                - rm -rf /tmp/hugo*
            build:
              commands:
                - hugo
            post_build:
              commands:
                - aws s3 cp --recursive --acl public-read ./public s3://${SiteBucketName}
          artifacts:
            type: zip
            files:
              - ./html/index.html
      TimeoutInMinutes: 10
  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      RoleArn: !GetAtt CodePipelineRole.Arn
      Stages:
      - Name: Source
        Actions:
        - InputArtifacts: []
          Name: Source
          ActionTypeId:
            Category: Source
            Owner: ThirdParty
            Version: '1'
            Provider: GitHub
          OutputArtifacts:
          - Name: SourceArtifacts
          Configuration:
            Owner: !Ref GitHubUser
            Repo: !Ref GitHubRepo
            Branch: !Ref GitHubBranch
            OAuthToken: !Ref GitHubToken
          RunOrder: 1
      - Name: Deploy
        Actions:
        - Name: Artifact
          ActionTypeId:
            Category: Build
            Owner: AWS
            Version: '1'
            Provider: CodeBuild
          InputArtifacts:
          - Name: SourceArtifacts
          OutputArtifacts:
          - Name: DeploymentArtifacts
          Configuration:
            ProjectName: !Ref CodeBuildDeploySite
          RunOrder: 1
      ArtifactStore:
        Type: S3
        Location: !Ref PipelineBucket
Outputs:
  PipelineUrl:
    Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
    Description: CodePipeline URL
  SiteUrl:
    Value: !GetAtt [SiteBucket, WebsiteURL]
    Description: S3 Website URL

EDIT 10/20

Still haven't solved this. I tried to follow the advice given below by matsev , but I'm still getting the same validation error. For completeness, the latest template I'm trying is

---

AWSTemplateFormatVersion: '2010-09-09'
Description: Pipeline using CodePipeline and CodeBuild for continuous delivery of a single-page application to S3
Parameters:
  SiteBucketName:
    Type: String
    Description: Name of bucket to create to host the website
  GitHubUser:
    Type: String
    Description: GitHub User
    Default: "stelligent"
  GitHubRepo:
    Type: String
    Description: GitHub Repo to pull from. Only the Name. not the URL
    Default: "devops-essentials"
  GitHubBranch:
    Type: String
    Description: GitHub Branch
    Default: "master"
  GitHubToken:
    NoEcho: true
    Type: String
    Description: Secret. It might look something like 9b189a1654643522561f7b3ebd44a1531a4287af OAuthToken with access to Repo. Go to https://github.com/settings/tokens
  BuildType:
    Type: String
    Default: "LINUX_CONTAINER"
    Description: The build container type to use for building the app
  BuildComputeType:
    Type: String
    Default: "BUILD_GENERAL1_SMALL"
    Description: The build compute type to use for building the app
  BuildImage:
    Type: String
    Default: "aws/codebuild/ubuntu-base:14.04"
    Description: The build image to use for building the app
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: "Site Configuration"
        Parameters:
          - SiteBucketName
      - Label:
          default: "GitHub Configuration"
        Parameters:
          - GitHubToken
          - GitHubUser
          - GitHubRepo
          - GitHubBranch
      - Label:
          default: "Build Configuration"
        Parameters:
          - BuildType
          - BuildComputeType
          - BuildImage
    ParameterLabels:
      SiteBucketName:
        default: Name of S3 Bucket to create for website hosting
      GitHubToken:
        default: GitHub OAuth2 Token
      GitHubUser: 
        default: GitHub User/Org Name
      GitHubRepo: 
        default: GitHub Repository Name
      GitHubBranch: 
        default: GitHub Branch Name
      BuildType: 
        default: CodeBuild type
      BuildComputeType: 
        default: CodeBuild instance type
      BuildImage: 
        default: CodeBuild image
Resources:
  CodeBuildRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - codebuild.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: codebuild-service
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action: "*"
            Resource: "*"
          Version: '2012-10-17'
  CodePipelineRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - codepipeline.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: codepipeline-service
        PolicyDocument:
          Statement:
          - Action:
            - codebuild:*
            Resource: "*"
            Effect: Allow
          - Action:
            - s3:GetObject
            - s3:GetObjectVersion
            - s3:GetBucketVersioning
            Resource: "*"
            Effect: Allow
          - Action:
            - s3:PutObject
            Resource:
            - arn:aws:s3:::codepipeline*
            Effect: Allow
          - Action:
            - s3:*
            - cloudformation:*
            - iam:PassRole
            Resource: "*"
            Effect: Allow
          Version: '2012-10-17'
  SiteBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
    Properties:
      AccessControl: PublicRead
      BucketName: !Ref SiteBucketName
      WebsiteConfiguration:
        IndexDocument: index.html
  PipelineBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete
  CodeBuildDeploySite:
    Type: AWS::CodeBuild::Project
    DependsOn: CodeBuildRole
    Properties:
      Name: !Sub ${AWS::StackName}-DeploySite
      Description: Deploy site to S3
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        Type: !Ref BuildType
        ComputeType: !Ref BuildComputeType
        Image: !Sub ${BuildImage}
        EnvironmentVariables:
          - Name: HUGO_VERSION
            Value: '0.49'
            Type: PLAINTEXT
      Source:
        Type: CODEPIPELINE
        BuildSpec: !Sub |
          version: 0.2
          env:
            variables:
              AWS_DEFAULT_REGION: "US-WEST-2"
          phases:
            install:
              commands:
                - printenv
                - curl -Ls https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
                - tar xf /tmp/hugo.tar.gz -C /tmp
                - mv /tmp/hugo_${HUGO_VERSION}_linux_amd64/hugo_${HUGO_VERSION}_linux_amd64 /usr/bin/hugo
                - rm -rf /tmp/hugo*
            build:
              commands:
                - hugo
            post_build:
              commands:
                - aws s3 cp --recursive --acl public-read ./samples s3://${SiteBucketName}/samples 
                - aws s3 cp --recursive --acl public-read ./html s3://${SiteBucketName}/ 
          artifacts:
            type: zip
            files:
              - ./html/index.html
      TimeoutInMinutes: 10
  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      RoleArn: !GetAtt CodePipelineRole.Arn
      Stages:
      - Name: Source
        Actions:
        - InputArtifacts: []
          Name: Source
          ActionTypeId:
            Category: Source
            Owner: ThirdParty
            Version: '1'
            Provider: GitHub
          OutputArtifacts:
          - Name: SourceArtifacts
          Configuration:
            Owner: !Ref GitHubUser
            Repo: !Ref GitHubRepo
            Branch: !Ref GitHubBranch
            OAuthToken: !Ref GitHubToken
          RunOrder: 1
      - Name: Deploy
        Actions:
        - Name: Artifact
          ActionTypeId:
            Category: Build
            Owner: AWS
            Version: '1'
            Provider: CodeBuild
          InputArtifacts:
          - Name: SourceArtifacts
          OutputArtifacts:
          - Name: DeploymentArtifacts
          Configuration:
            ProjectName: !Ref CodeBuildDeploySite
          RunOrder: 1
      ArtifactStore:
        Type: S3
        Location: !Ref PipelineBucket
Outputs:
  PipelineUrl:
    Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
    Description: CodePipeline URL
  SiteUrl:
    Value: !GetAtt [SiteBucket, WebsiteURL]
    Description: S3 Website URL

Answer 1

I believe the following happens: CloudFormation attempts to resolve ${HUGO_VERSION} as a Parameter of Cloudformation Template as it is within the "!Sub" function.

From AWS documentation on Sub function

To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html#w2ab1c21c28c59b7

Therefore your build spec phase should be similar to:

      phases:
        install:
          commands:
            - printenv
            - echo "Install step..."
            - curl -Ls https://github.com/gohugoio/hugo/releases/download/v${!HUGO_VERSION}/hugo_${!HUGO_VERSION}_Linux-64bit.tar.gz -o /tmp/hugo.tar.gz
            - tar xf /tmp/hugo.tar.gz -C /tmp
            - mv /tmp/hugo_${!HUGO_VERSION}_linux_amd64/hugo_${!HUGO_VERSION}_linux_amd64 /usr/bin/hugo
            - rm -rf /tmp/hugo*

Hope this helps!

Answer 2

Please check the Environment property of the AWS::CodeBuild::Project in your CloudFormation template. Specifically the EnvironmentVariables allows you to specify environment variables, eg

  CodeBuildDeploySite:
    Type: AWS::CodeBuild::Project
    DependsOn: CodeBuildRole
    Properties:
      Name: !Sub ${AWS::StackName}-DeploySite
      Description: Deploy site to S3
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        Type: !Ref BuildType
        ComputeType: !Ref BuildComputeType
        Image: !Sub ${BuildImage}
        EnvironmentVariables:
          - Name: HUGO_VERSION
            Value: '0.49'
            Type: PLAINTEXT

# More properties...

Now in you can reference the HUGO_VERSION as an environment variable in your buildspec.yml file, eg

pre_build:    
  commands:
    - echo HUGO_VERSION $HUGO_VERSION