Terraform project structure

Recently I gave a terraform talk, and people ask how to get started with terraform. The simple answer is to try to get your hands dirty with it, but at certain point you have to learn how to "manage" the ever-growing terraform project.

Single-project structure

In the beginning, you can probably make do with a single terraform workspace containing all resources. A single terraform project can look like this:

.
├── backend.tf
├── main.tf
├── outputs.tf
├── providers.tf
├── terraform.tf
└── variables.tf

backend.tf

This is where you store your terraform state.

terraform {
  backend "gcs" {
    bucket = "xxx"
    prefix = "terraform/xxx"
  }
}

main.tf

Your terraform resources should be defined here. You can split it into multiple files for each resource/service group.

resource "google_cloud_run_v2_job" "this" {
  // define your resource here
}

outputs.tf

Anything you want to export stays here.

output "foo" {
  value = "xxx"
}

providers.tf

This is where you define required terraform providers. You probably have to supply required values to authenticate to services.

provider "google" {
  project = var.project_id
  region  = var.region
}

provider "sops" {}

terraform.tf

This file is equivalent to your project's dependencies - what your project requires to successfully create and modify resources.

terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "6.2.1"
    }
    google = {
      source  = "hashicorp/google"
      version = "5.23.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.5.1"
    }
    sops = {
      source  = "carlpett/sops"
      version = "1.0.0"
    }
  }
}

variables.tf

Anything specific to your setup and can be changed should be defined as variables.

variable "project_id" {
  type    = string
  default = "xxx"
}
variable "region" {
  description = "The region of resources"
  type        = string
  default     = "asia-southeast1"
}

Sometimes the variables contain secrets, in which case you might want to create a separate terraform.tfvars and specify your values there. Don't check this file into git.

teraform.tfvars should look like this:

foo = "bar"

Multi-project structure

Over time your terraform project can contain more resources - taking longer and longer to terraform plan and terraform apply. If you start experiencing this, it is probably time to split your current project into multiple projects. There are plenty of resources online for this, but I would suggest you to practice it before splitting your production project, as refactoring a terraform project is a very delicate process.

Over the years I have experimented with a lot of terraform project structure, since I had to maintain all infrastructure in an organization, spanning all products they have. Following structure has been serving me well:

.
└── environments
    ├── core
    │   ├── gcp
    │   │   ├── logging
    │   │   └── networking
    │   └── github
    ├── dev
    │   ├── aws
    │   │   ├── productA
    │   │   └── productB
    │   └── gcp
    │       ├── productC
    │       └── productD
    ├── non-prod
    │   ├── aws
    │   │   ├── playgroundA
    │   │   └── playgroundB
    │   └── gcp
    │       ├── sandboxA
    │       └── sandboxB
    └── prod
        ├── aws
        │   ├── productA
        │   └── productB
        └── gcp
            ├── productC
            └── productD

Essentially, you separate the root level by environments. Under each environment, separate each infrastructure/service vendor. Finally, group terraform resources by product/project.

You'll notice that for core infrastructure it has its own core environment, because it is shared between almost all environments (for instance, shared logging sink for all environments).