Best practices when using Terraform [closed]

Answer 1

Some Terraform Best Practices to Follow:

1. Avoid hard coding: Sometimes developers manually created resources directly. You need to mark these resource and use terraform import to include them in codes. A sample:

   account_number=“123456789012" account_alias="mycompany"

2. Run Terraform from a docker container: Terraform releases an official Docker container that allows you to easily control which version you can run.

It is recommended to run the Terraform Docker container when you set your build job in the CI/CD pipeline.

TERRAFORM_IMAGE=hashicorp/terraform:0.11.7
TERRAFORM_CMD="docker run -ti --rm -w /app -v ${HOME}/.aws:/root/.aws -v ${HOME}/.ssh:/root/.ssh -v `pwd`:/app $TERRAFORM_IMAGE"

For more, please refer to my blog: https://medium.com/tech-darwinbox/how-darwinbox-manages-infrastructure-at-scale-with-terraform-371e2c5f04d3

Answer 2

I know there’s a lot of answers here but my approach is quite different.

⁃   Modules
⁃   Environment management 
⁃   Separation of duties

Modules

1. Create modules for logical collections of resources. Example: If your goal is to deploy an API, which requires a DB, HA VMs, autoscaling, DNS, PubSub and object storage then all of these resources should be templated in a single module.
2. Avoid creating modules that utilise a single resource. This can and has been done and a lot of the modules in the registry do this but it’s a practice that helps with resource accessibility rather than infrastructure orchestration. Example: A module for AWS EC2 helps the user access the EC2 by making complex configurations more simple to invoke but a module like the example in 1. assists the user when orchestrating application, component or service driven infrastructure.
   3. Avoid resource declarations in your workspace. This is more about keeping your code tidy and organised. As modules are easily versioned, you have more control over your releases.

Environment management

IaC has made SDLC process relevant to infrastructure management and it’s not normal to expect to have development infrastructure as well as development application environments.

1. Don’t use folders to manage your IaC environments. This leads to drift as there’s no common template for your infrastructure.
2. Do use a single workspace and variables to control environment specifications. Example: Write your modules so that when you change the environment variable (var.stage is popular) the plan alters to fit your requirements. Typically the environments should vary as little as possible with quantity, exposure and capacity usually being the variable configurations. Dev might deploy 1 VM with 1 core and 1GB RAM in private topology but production may be 3 VMs with 2 cores and 4GB RAM with additional public topology. You can of course have more variation: dev may run database process on the same server as the application to save cost but production may have a dedicated DB instance. All of this can be managed by changing a single variable, ternary statements and interpolation.

Separation of duties

If you’re in a small organisation or running personal infrastructure this doesn’t really apply but it will help you manage your operations.

1. Break down your infrastructure by duties, responsibilities or teams. Example: Central IT control underlying shared services (virtual networks, subnets, public IP addresses, log groups, governance resources, multi tenanted DBs, shared keys, etc.) whilst the API team only control the resources needed for their service (VMs, LBs, PubSub etc) and consume Central ITs services through data source and remote state lookups.
   2. Govern team access. Example: Central IT may have admin rights but the API team only have access to a restricted set of public cloud APIs.

This also helps with release concerns as you will find some resources rarely change whilst others change all the time. Separation removes risk and complexity.

This strategy draws parallels with AWS’ multi account strategy. Have a read for more info.

CI/CD

This is a topic of its own but Terraform works very well within a good pipeline. The most common error here is to treat CI as a silver bullet. Technically Terraform should only be provisioning infrastructure during stages of an assembly pipeline. This would be separate to what happens in CI stages where one typically validates and tests the templates.

N.B. Written on mobile so please excuse any errors.

Answer 3

Previously remote config allowed this but now has been replaced by "backends", so terraform remote is not anymore available.

terraform remote config -backend-config="bucket=<s3_bucket_to_store_tfstate>" -backend-config="key=terraform.tfstate" -backend=s3
terraform remote pull
terraform apply
terraform remote push

See the docs for details.

Answer 4

Before answers have been very solid and informative, I will try to add my 2 cents here

Common recommendations for structuring code

1. It is easier and faster to work with smaller number of resources:

   • Cmdsterraform plan and terraform apply both make cloud API calls to verify the status of resources.
   • If you have your entire infrastructure in a single composition this can take many minutes (even if you have several files in the same folder).

2. Blast radius is smaller with fewer resources:

   • Insulating unrelated resources from each other by placing them in separate compositions (folders) reduces the risk if something goes wrong.

3. Start your project using remote state:

   • Your laptop is no place for your infrastructure source of truth.
   • Managing a tfstate file in git is a nightmare.
   • Later when infrastructure layers starts to grow in any direction (number of dependencies or resources).
   • example module: https://github.com/cloudposse/terraform-aws-tfstate-backend
   • ref tool: https://github.com/camptocamp/terraboard

4. Try to practice a consistent structure and naming convention:

   • Like procedural code, Terraform code should be written for people to read first, consistency will help when changes happen six months from now.
   • It is possible to move resources in Terraform state file but it may be harder to do if you have inconsistent structure and naming.

5. Keep resource modules as plain as possible.

6. Don't hard-code values which can be passed as variables or discovered using data sources.

7. Use data sources and terraform_remote_state specifically as a glue between infrastructure modules within composition.

(ref article: https://www.terraform-best-practices.com/code-structure)

---

Example:

It is easier and faster to work with smaller number of resources so below we present a recommended code layout.

NOTE: just as reference not to be strictly follow since each project has it's own specific characteristics

.
├── 1_tf-backend #remote AWS S3 + Dynamo Lock tfstate 
│   ├── main.tf
│   ├── ...
├── 2_secrets
│   ├── main.tf
│   ├── ...
├── 3_identities
│   ├── account.tf
│   ├── roles.tf
│   ├── group.tf
│   ├── users.tf
│   ├── ...
├── 4_security
│   ├── awscloudtrail.tf
│   ├── awsconfig.tf
│   ├── awsinspector.tf
│   ├── awsguarduty.tf
│   ├── awswaf.tf
│   └── ...
├── 5_network
│   ├── account.tf
│   ├── dns_remote_zone_auth.tf
│   ├── dns.tf
│   ├── network.tf
│   ├── network_vpc_peering_dev.tf
│   ├── ...
├── 6_notifications
│   ├── ...
├── 7_containers
│   ├── account.tf
│   ├── container_registry.tf
│   ├── ...
├── config
│   ├── backend.config
│   └── main.config
└── readme.md

Answer 5

Use terraform cloud for manage and save states, together with advises above.

Answer 6

I believe there are few best practices need to follow while using terraform for orchestrating the infrastructure

1. Don't write the same code again ( Reusability)
2. Keep environment configuration separate to maintain it easily.
3. Use remote backend s3(encrypted) and dynamo DB to handle the concurrency locking
4. Create a module and use that module in main infrastructure multiple time, its like a reusable function which can be called multiple time by passing different parameter.

Handle multiple environments

Most of the time recommended way is to use terraform 'workspace' to handle the multiple environments but I believe the usage of workspace could vary based on way of work in an organization. Other is storing the Terraform code for each of your environments (e.g. stage, prod, QA) to separate the environment states. However, in this case we are just copying the same code at many places.

├── main.tf
├── dev
│   ├── main.tf
│   ├── output.tf
│   └── variables.tf
└── prod
├── main.tf
├── output.tf
└── variables.tf

I followed some different approach to handle and avoid the duplication of the same terraform code by keeping in each environment folder since I believe most of the time all environment would be 90% same.

├── deployment
│ ├── 01-network.tf
│ ├── 02-ecs_cluster.tf
│ ├── 03-ecs_service.tf
│ ├── 04-eks_infra.tf
│ ├── 05-db_infra.tf
│ ├── 06-codebuild-k8s.tf
│ ├── 07-aws-secret.tf
│ ├── backend.tf
│ ├── provider.tf
│ └── variables.tf
├── env
│ ├── dev
│ │ ├── dev.backend.tfvar
│ │ └── dev.variables.tfvar
│ └── prod
│ ├── prod.backend.tfvar
│ └── prod.variables.tfvar
├── modules
│ └── aws
│ ├── compute
│ │ ├── alb_loadbalancer
│ │ ├── alb_target_grp
│ │ ├── ecs_cluster
│ │ ├── ecs_service
│ │ └── launch_configuration
│ ├── database
│ │ ├── db_main
│ │ ├── db_option_group
│ │ ├── db_parameter_group
│ │ └── db_subnet_group
│ ├── developertools
│ ├── network
│ │ ├── internet_gateway
│ │ ├── nat_gateway
│ │ ├── route_table
│ │ ├── security_group
│ │ ├── subnet
│ │ ├── vpc
│ └── security
│ ├── iam_role
│ └── secret-manager
└── templates

Configuration related to environments

Keep environment related configuration and parameters separate in a variable file and pass that value to configure the infrastructure. e.g as below

• dev.backend.tfvar

    region = "ap-southeast-2"
    bucket = "dev-samplebackendterraform"
    key = "dev/state.tfstate"
    dynamo_db_lock = "dev-terraform-state-lock"
  

• dev.variable.tfvar

  environment                     =   "dev"
  vpc_name                        =   "demo"
  vpc_cidr_block                  =   "10.20.0.0/19"
  private_subnet_1a_cidr_block    =   "10.20.0.0/21"
  private_subnet_1b_cidr_block    =   "10.20.8.0/21"
  public_subnet_1a_cidr_block     =   "10.20.16.0/21"
  public_subnet_1b_cidr_block     =   "10.20.24.0/21"
  

Conditional skipping of infrastructure part

Create a configuration in env specific variable file and based on that variable decide to create or skipping that part. In this way based on need the specific part of the infrastructure can be skipped.

variable vpc_create {
   default = "true"
}

module "vpc" {
  source = "../modules/aws/network/vpc"
  enable = "${var.vpc_create}"
  vpc_cidr_block = "${var.vpc_cidr_block}"
  name = "${var.vpc_name}"
 }

 resource "aws_vpc" "vpc" {
    count                = "${var.enable == "true" ? 1 : 0}"
    cidr_block           = "${var.vpc_cidr_block}"
    enable_dns_support   = "true"
   enable_dns_hostnames = "true"
}

below command is required to initialize and execute the infra changes for each environment, cd to the required environment folder.

  terraform init -var-file=dev.variables.tfvar -backend-config=dev.backend.tfvar ../../deployment/

  terraform apply -var-file=dev.variables.tfvar ../../deployment

For reference: https://github.com/mattyait/devops_terraform