如何告诉teamcity代理，哪个aws账户运行terraform到

Question

I spun up TeamCity into a Kube.netes cluster. I have one deployment for the TeamCity server and one deployment for the TeamCity build agent. When I run Terraform on the TeamCity build agent, it creates resources in the same AWS account as the EC2 instance which is hosting TeamCity Kube.netes cluster.

I would like to run a build, which creates AWS resources in a separate AWS account. My thinking is to assign an AWS access key and secret key to the TeamCity server and pass them to the TeamCity build agent, but I don't know how the work flow is going to be.

Currently, I have declared AWS access key and secret keys as environment variables on the build but they are not getting passed to agent. My build steps only contains 3 lines

terraform init
terraform plan
terraform apply -auto-approve

Answer 1

I dealt with a very similar situation. When we started using AWS, we did not follow best practice around how our scripts authenticate to AWS.

We created users in each account and exported the access and secret keys into our TeamCity build automation to allow our scripts to run in the context of these accounts.

You will wind up with several users in different accounts and will wind up manually rotating the keys for these users, then updating the access and secret keys in TeamCity everytime you need to rotate the keys.

As the TeamCity build agent servers are EC2 instances, it is best practice in AWS to use roles instead of users in situations like this. From the AWS documentation:

Applications that run on an Amazon EC2 instance need credentials in order to access other AWS services. To provide credentials to the application in a secure way, use IAM roles. A role is an entity that has its own set of permissions, but that isn't a user or group. Roles also don't have their own permanent set of credentials the way IAM users do. In the case of Amazon EC2, IAM dynamically provides temporary credentials to the EC2 instance, and these credentials are automatically rotated for you.

Source: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#use-roles-with-ec2

What you should do in this situation is create a role in each account that has access to create the resources that terraform will build. You will need another role for the TeamCity build agent servers, this role should be able to assume the roles created in each account.

Let's say you have two accounts:

1. AccountA is where you want the build agents to run build resources with terraform.
2. AccountB is the account your build agents are running from.

What you will need is:

1. A role in AccountA with access to create resources.
2. A role in AccountB that grants access for ec2 instances to assume the above role.

The instance profile of the role in AccountB should be assigned to the TeamCity build agent servers. The build agent servers are then able to assume the role in AccountA and run in the context of that account with the permissions they need without the need for hard-coded access and secret keys.

You can repeat this process for as many other accounts as you want to build resources in through Terraform.

You will have to add the following block to the aws provider in your terraform scripts to let terraform know what role to assume:

provider "aws" {
assume_role {
role_arn = "arn:aws:iam::<ACCOUNTA>:role/<ROLENAME>"
}
}

Here is some AWS documentation on how to grant access to an S3 bucket in a different account from an EC2 instance: https://aws.amazon.com/premiumsupport/knowledge-center/s3-instance-access-bucket/

Another useful documentation page on that covers IAM Roes, EC2 instance profiles & cross account access via Roles is here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

One last link goes through exactly what you are trying to do: https://blog.e-zest.com/aws-ec2-assume-role-with-terraform

Answer 2

My agent.yml file looks like this when i want to add two annotations for 2 separate accounts

spec: replicas: 1 selector: matchLabels: app: agent-pod template: metadata: annotations: iam.amazonaws.com/role: arn:aws:iam::account:role/accoun1-s3-role iam.amazonaws.com/role: arn:aws:iam::account:role/account2-pods-s3-role

but as soon as i put two annotation, my sts assume role for pod only shows me second role. and when i run the build from teamcity i get this error no valid credential sources found for aws provider

Answer 3

Security best practices regarding TeamCity + AWS org with multiple accounts:

• Use iam roles instead of user accounts with API keys with least priviledges possible and attach it as an agent instance profile
• TeamCity agent is usually in a single agent/multi AWS accounts deployments shared across multiple projects. That means, anybody can gain an access to a multiple accounts easily. A solution - either build a dedicated agent in every account with least priviledge instance policy/IAM role attached and connected trough for ex. transit gw back to team-city server, or if you have a dedicated CI/CD account, build dedicated agents for each account, come up with a tagging policy and create cross account IAM roles with additional ABAC policy so only the specific instance will able to assume a role in the remote account
• secure your CI/CD as much as you can, use least account priviledges, enforce a good network policy, use tools like edr, siem etc. - !!! this is one of your most vulnerable asset with a very wide attack surface in your organization !!!
• avoid tools like Jenkins, if you search for CVEs, it's like a swiss cheese