Running `terraform destroy` command daily for dev environments

Question

Just wondering what is the best practice here.

I have these dev environments, dev/QA/UAT/ab/monkey and so on, which are used only during the daytime. We would like to save some cost here, by shutting them down during nighttime.

Each environment consists of frontend/API/caching/queueing/DB servers/Docker images.

Is using Terraform's create/destroy daily the right approach here?

First thing I noticed is the IP address change on removing EC2 instances. Every day on destroying the env, I will have to re-map the DNS. But this can be solved using EC2 elastic IP. But then I read somewhere:

if you're using an EIP to just provide a public IP and not to rapidly and seamlessly distribute traffic in the event of an outage while keeping DNS records the same, it's best to just use the AWS non-EIP pub IP and DNS records for pub access

Does AWS give a public DNS that doesn't go away if I shut down the EC2 instance?

Next is of course the data back-ups that I have to do. I have to back up all DBs, assets like images and videos, logs are not a concern since I will be pushing them off to another server using a log collector agent but all other data needs to be backed up before removal using Terraform destroy. I will also have tones of ECR images, I guess I need to back up them as well.

This feels like a lot of work. What is the best practice?

Just to add, almost all environments will run through-out the year.

Answer 1

You definitely could destroy these environments every day, depending on where your infrastructure as code lives, you could do this in a number of ways. For example if it's in a github repo, using github actions and workflows, you could create a task that runs a little while after you finish each day that would destroy everything. Other options would be gitlab which has it's own way of doing this, or something like Jenkins/TeamCity/Bamboo/CircleCI which could automate the job for you.

In theory you could set up another job that applies them again each weekday morning, so you can save money and you don't waste time each morning setting up your dev envs.

With regards to your DNS issues, if you are managing your DNS records with route53, you can add a resource for your records which point to the public IP of your instance (that would be an A record), or the public DNS of your instance (for example). Then when you create the new resources each morning the records will be updated to point at your new instances.

Simply shutting down the instances isn't always going to cut all of your costs, as you will still be paying for some resources like the EBS volumes and if you have elastic IPs which are not in use you get charged for that, load balancers generate charges even when not in use etc