Infrastructure as code in practice: structuring Terraform for a team that is growing

Terraform problems are almost never Terraform problems. They are boundary problems: one state file that everything writes to, a module that knows too much about its caller, an environment that drifted because somebody fixed production by hand at 2am.

This is the structure we use and the reasoning for each boundary.

State boundaries decide everything else

Before folders, before modules: decide what shares a state file. That single choice determines your blast radius, your plan times and who can break what.

Split along two axes — by environment, and by rate of change:

state/
  prod/network         rarely changes, everything depends on it
  prod/data            databases, buckets — changes carefully
  prod/platform        cluster, load balancers
  prod/services/api    changes several times a week
  staging/...          same shape

Cross-boundary references go through data sources or explicit outputs, never by reaching into another layer's state:

data "aws_vpc" "main" {
  tags = { Name = "prod-main" }
}

That indirection is the point. The service layer knows a VPC with that tag exists; it does not know or care how the network layer built it, and cannot accidentally modify it.

A module is an interface, not a folder

The most common mistake is turning every group of resources into a module. A module earns its place when it hides a decision — otherwise it is indirection with a variables file.

A good module has a narrow, opinionated interface:

module "service" {
  source = "../../modules/ecs-service"
 
  name          = "api"
  image         = var.image          # the immutable tag from CI
  cpu           = 512
  memory        = 1024
  desired_count = 3
 
  # The module decides: log group, retention, task role, target group,
  # health check, deployment circuit breaker, alarms.
}

Six inputs, and it produces perhaps thirty resources with the conventions already applied. Compare that to a module taking forty variables, which is the same resources with an extra file in front of them.

1. Version modules, and pin the version at the call site

   A module in a separate repository at ?ref=v2.3.0. Otherwise a change to a shared module changes every environment at once, and you find out during the next apply.

2. Never put a provider block inside a module

   It makes the module impossible to use across regions or accounts, and removing it later is a breaking change for every caller.

3. Output what callers need, not everything

   Every output is part of the interface. A module returning its whole resource set has no interface.

Environments differ by variables, not by copies

The failure that produces drift is copy-pasting an environment folder and editing it. Six months later staging and production differ in ways nobody can enumerate — which means staging has stopped testing anything.

environments/
  prod/main.tf        ← identical composition
  prod/prod.tfvars    ← the differences, in one readable file
  staging/main.tf     ← identical composition
  staging/staging.tfvars

Making the plan reviewable

A plan output nobody reads is a review that is not happening. Three things make it readable.

1. Post the plan on the pull request, and require an apply after merge

   Plan on PR, apply on merge to main. The change is reviewed as a diff of proposed infrastructure rather than as a diff of HCL.

2. Make destructive changes impossible to miss

   Grep the plan for replacements and deletions and label the PR. must be replaced on a database is the line that matters, and it is easy to miss in four hundred lines of output.

3. Keep plans small by keeping states small

   A plan touching six resources is read. A plan touching two hundred is scrolled past. This is the practical payoff of the state boundaries above.

# For the resources where a replacement is never acceptable.
resource "aws_db_instance" "main" {
  # …
  lifecycle {
    prevent_destroy = true
  }
}

Drift is a process problem

Someone will change something in the console during an incident. That is not a discipline failure — it is what you want them to do when the site is down. The failure is not reconciling it afterwards.

1. Run a scheduled plan and alert on any diff

   Nightly, against every environment. A non-empty plan on a day nobody deployed is drift, and you want to hear about it the next morning rather than the next quarter.

2. Import rather than recreate

   Whatever was made by hand, import it into state. Deleting and re-creating is how a console fix becomes an outage.

3. Treat repeated drift in one place as a design signal

   If people keep changing the same thing by hand, it probably should not be in Terraform — or it needs a safer path.

Secrets never go in state

Terraform state contains resource attributes in plaintext, including generated passwords. That is not a bug you can configure away.

1. Encrypt the state backend and restrict access to it

   State is a credential store whether or not you intended it to be.

2. Create the secret container, not the secret

   Terraform makes the Secrets Manager entry; a separate process populates the value. The application reads it at runtime.

3. Reference secrets by ARN in task definitions

   So the value is resolved by the runtime and never passes through a plan.

What to enforce in CI

1. fmt and validate on every commit

   Formatting arguments in review are wasted review.

2. A static analyser for security misconfiguration

   Public buckets, open security groups, unencrypted volumes — the findings are usually real and usually one line to fix.

3. A policy check for the rules you actually care about

   Every resource tagged with an owner and an environment; no 0.0.0.0/0 on anything but a load balancer. Encode the rules rather than repeating them in review.

4. Cost estimation on the plan

   A PR that adds an instance type nobody priced is worth catching before the invoice.

The shape, summarised

1. Split state by environment and rate of change, before anything else

   It determines blast radius, plan time and how many people can work at once.

2. Write a module only when it hides a decision

   Narrow interface, versioned, pinned at the call site, no provider block inside.

3. One composition per environment; differences live in tfvars

   Divergent main.tf files are how staging quietly stops testing production.

4. Plan on the PR, apply on merge, flag replacements loudly

   And keep plans small enough that somebody reads them.

5. Detect drift nightly, import rather than recreate

   A console fix during an incident is correct; not reconciling it is not.