Terraform at Scale: Modules, Remote State, and the Drift Problem

May 23, 2026 21 min read DevOps

terraformdevopsinfrastructure-as-codeawscloudopen-source

Contents

Terraform is easy to start with and surprisingly hard to scale. A single main.tf that creates a VPC and a few EC2 instances works fine on a laptop. Two engineers working on the same infrastructure, three environments (dev/staging/prod), twenty modules shared across five teams, and a CI/CD pipeline that needs to run terraform apply without a human watching — that is a different problem. The tooling supports it, but the defaults get you there slowly and painfully.

This post covers the non-obvious parts of Terraform at scale: remote state that does not race, environment separation that does not duplicate, modules that do not silently break production when updated, and a practical approach to drift — the gap between what Terraform thinks exists and what actually exists in your cloud account.

It also covers the mechanics of refactoring Terraform configurations without destroying and recreating resources, and what to do when state gets out of sync in ways that terraform plan cannot fix automatically.

Remote State: S3 Backend

Local state (terraform.tfstate on disk) is fine for solo projects. For anything shared, it is a liability: it cannot be locked, it lives on whoever’s laptop last ran apply, and it will be lost or corrupted. Move to remote state before the first second engineer joins.

The S3 backend stores state in an S3 bucket. Since Terraform v1.10 (AWS provider v5+), it supports native S3 locking via conditional writes — no DynamoDB table required. DynamoDB-based locking is deprecated and will be removed in a future version.

Bootstrap the backend resources

The bootstrap is the annoying part: you need S3 and (previously) DynamoDB to exist before you can use them as a backend, but if you manage them with Terraform, where does their state go? The answer is a dedicated bootstrap configuration managed outside the main stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


# bootstrap/main.tf
# Run once manually (terraform apply), then never touch again.
# State for this configuration lives locally; it is low-risk since
# the resources are stable and changes are rare.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
}

resource "aws_s3_bucket" "tfstate" {
  bucket = "acme-terraform-state-prod"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "tfstate" {
  bucket                  = aws_s3_bucket.tfstate.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Backend configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# backend.tf (in each stack)
terraform {
  backend "s3" {
    bucket       = "acme-terraform-state-prod"
    key          = "networking/vpc/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true

    # Native S3 locking (Terraform >= 1.10, AWS provider >= 5.0)
    use_lockfile = true
  }
}

The key is the S3 object path for the state file. A consistent naming convention here prevents state collisions and makes it obvious which stack owns each file:

<team>/<service>/<component>/terraform.tfstate
networking/vpc/terraform.tfstate
platform/eks/cluster/terraform.tfstate
data/rds/postgres/terraform.tfstate
apps/payments/ecs/terraform.tfstate

Versioning on the bucket means every state write creates a new version. If an apply corrupts state (it happens), you can roll back by restoring a previous version from the S3 console or via CLI.

OIDC authentication in CI

Do not use long-lived IAM access keys in CI. Use OIDC federation so GitHub Actions (or GitLab CI, or CircleCI) can assume an IAM role via a short-lived token:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# iam.tf — trust policy for GitHub Actions
data "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"
}

resource "aws_iam_role" "terraform_ci" {
  name = "terraform-ci"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = data.aws_iam_openid_connect_provider.github.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringLike = {
          "token.actions.githubusercontent.com:sub" = "repo:acme-org/*:*"
        }
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

In the GitHub Actions workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/terraform-ci
      aws-region: us-east-1

  - run: terraform init && terraform plan

The credentials last 15 minutes by default and are scoped to the role’s permissions. No secrets to rotate, no leaked keys in CI logs.

Environment Separation: Workspaces vs Directories

Terraform workspaces partition a single backend key into named slots (default, dev, staging, prod). They look attractive as an environment mechanism but have a significant flaw: all workspaces share the same configuration. You cannot use workspace A with Terraform version X and workspace B with version Y, and you cannot easily have different provider configurations, different module versions, or different variable defaults per workspace without writing brittle conditional logic.

The workspace-as-environment pattern leads to constructs like this, which is a sign something is wrong:

1
2
3
4


# Anti-pattern: workspace-conditional sizing
resource "aws_instance" "app" {
  instance_type = terraform.workspace == "prod" ? "m6i.2xlarge" : "t3.micro"
}

The industry-standard alternative is directory-based environment separation: each environment is a directory with its own backend configuration, its own terraform.tfvars, and references to versioned modules:

infrastructure/
├── modules/
│   ├── vpc/
│   ├── eks/
│   └── rds/
└── environments/
    ├── dev/
    │   ├── backend.tf       # key = "dev/..."
    │   ├── main.tf          # module sources pinned to a version
    │   └── terraform.tfvars
    ├── staging/
    │   ├── backend.tf       # key = "staging/..."
    │   ├── main.tf
    │   └── terraform.tfvars
    └── prod/
        ├── backend.tf       # key = "prod/..."
        ├── main.tf
        └── terraform.tfvars

Each environment directory runs terraform init/plan/apply independently. Promoting a change from dev to prod is a diff in main.tf or terraform.tfvars, reviewed in a pull request, applied by CI.

Workspaces have a legitimate use: ephemeral per-developer sandboxes, or feature branch environments that are identical to the base configuration and created/destroyed automatically. For long-lived environments, use directories.

Module Architecture

Local vs registry vs git-sourced

Terraform supports four module source types:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# Local path — used during development, not for shared modules
module "vpc" {
  source = "../modules/vpc"
}

# Terraform Registry — public or private
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.8"
}

# Git with a tag — explicit, auditable, the right choice for internal shared modules
module "vpc" {
  source = "git::https://github.com/acme-org/terraform-modules.git//vpc?ref=v2.3.1"
}

# Git with a SHA — pinned absolutely, no ambiguity
module "vpc" {
  source = "git::https://github.com/acme-org/terraform-modules.git//vpc?ref=abc1234"
}

For internal modules shared across teams, git tags with semantic versioning is the right model. It gives you:

Stability: production environments pin to a released tag; updating requires an explicit change in main.tf
Rollback: reverting a module update is a one-line change
Audit trail: the PR that updates a module reference documents the intent and reviewer

The monorepo vs multi-repo question: if all your modules live in one repository, git tags apply to the whole repo, which means v2.3.1 of the VPC module and v2.3.1 of the EKS module are the same tag. This is workable if modules are versioned together (common in small teams) but problematic at scale. The Terraform Private Registry requires one module per repository. Separate module repos enable independent versioning but increase repository sprawl. There is no universally correct answer; the split becomes necessary when different teams own different modules at different cadences.

Module structure

A module is a directory with at minimum main.tf, variables.tf, and outputs.tf. A well-structured internal module:

modules/vpc/
├── main.tf            # Resources
├── variables.tf       # Input declarations with types and descriptions
├── outputs.tf         # Output values consumers need
├── versions.tf        # required_providers and version constraints
└── README.md          # Usage examples and variable reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# versions.tf — pin the provider constraint in the module
terraform {
  required_version = ">= 1.9"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0, < 6.0"
    }
  }
}

Module version constraints use pessimistic constraint operators. ~> 5.8 means >= 5.8, < 6.0. ~> 5 means >= 5.0, < 6.0. Avoid >= 5.0 without an upper bound — major version bumps break provider APIs and you do not want production to pick them up silently.

Avoiding module sprawl

The opposite failure mode from no modules is too many modules with too many layers of abstraction. A module that wraps a single resource and adds three variables adds overhead without value. A useful module encapsulates a coherent subsystem (a VPC with its subnets, route tables, and gateways; an EKS cluster with its node groups and IAM roles) where the internal complexity would otherwise be repeated.

The question to ask before creating a module: is the complexity I am hiding actually repeated elsewhere, or am I abstracting for abstraction’s sake? Three similar resource blocks is not a module candidate. Three identical stacks across three environments is.

Terragrunt: DRY Configuration at Scale

Terragrunt is a thin wrapper around Terraform that solves the repetition problem in directory-based environment management. The core problem it addresses: every environment directory needs an identical backend.tf with slightly different values (the key path), and an identical provider.tf, and references to the same module versions. With ten environments across five teams, that is fifty backend files to keep synchronized.

Terragrunt consolidates this into an root.hcl at the top of the tree that all environment configurations inherit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


# root.hcl (at repo root)
locals {
  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  env_vars     = read_terragrunt_config(find_in_parent_folders("env.hcl"))

  account_id = local.account_vars.locals.account_id
  env        = local.env_vars.locals.environment
  region     = "us-east-1"
}

generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    terraform {
      backend "s3" {
        bucket       = "acme-terraform-state-${local.account_id}"
        key          = "${path_relative_to_include()}/terraform.tfstate"
        region       = "${local.region}"
        encrypt      = true
        use_lockfile = true
      }
    }
  EOF
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    provider "aws" {
      region = "${local.region}"
      default_tags {
        tags = {
          Environment = "${local.env}"
          ManagedBy   = "terraform"
        }
      }
    }
  EOF
}

Each environment leaf is then minimal:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# environments/prod/networking/vpc/terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "git::https://github.com/acme-org/terraform-modules.git//vpc?ref=v2.3.1"
}

inputs = {
  cidr_block           = "10.0.0.0/16"
  availability_zones   = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnet_cidrs  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
  enable_nat_gateway   = true
}

Terragrunt also handles inter-stack dependencies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# environments/prod/eks/terragrunt.hcl
dependency "vpc" {
  config_path = "../networking/vpc"
  mock_outputs = {
    vpc_id          = "vpc-00000000"
    private_subnets = ["subnet-00000000"]
  }
  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}

inputs = {
  vpc_id          = dependency.vpc.outputs.vpc_id
  private_subnets = dependency.vpc.outputs.private_subnets
}

The mock_outputs block lets terraform plan run without executing the dependency — useful in CI when you want to validate a change without running the full dependency chain.

Run across an entire environment at once:

1
2
3
4
5
6
7
8


# Plan everything in prod
terragrunt run-all plan --terragrunt-working-dir environments/prod

# Apply with dependency ordering automatically resolved
terragrunt run-all apply --terragrunt-working-dir environments/prod

# Apply a single stack
terragrunt apply --terragrunt-working-dir environments/prod/eks

The Drift Problem

Drift is the divergence between the infrastructure Terraform’s state believes exists and what actually exists in the cloud. It is inevitable. An engineer manually scales an Auto Scaling Group during an incident. A compliance tool adds tags to resources. A cloud-managed service updates its own configuration. A resource gets deleted outside Terraform.

The question is not whether drift will happen but how quickly you detect it and what you do about it.

Detecting drift

terraform plan detects drift — if the plan shows changes that should not exist given recent apply history, that is drift. The problem is that terraform plan is only run when someone runs it. Scheduled drift detection runs plans on a cron schedule and alerts on unexpected output.

A minimal GitHub Actions workflow for scheduled drift detection:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


# .github/workflows/drift-detection.yml
name: Drift Detection

on:
  schedule:
    - cron: "0 6 * * *"   # Daily at 6 AM UTC
  workflow_dispatch:        # Allow manual trigger

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      issues: write

    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.9.x"

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.TERRAFORM_CI_ROLE_ARN }}
          aws-region: us-east-1

      - name: Plan all stacks
        id: plan
        run: |
          set +e
          DRIFT_FOUND=false
          REPORT=""

          for dir in environments/prod/*/; do
            cd "$dir"
            terraform init -input=false -no-color > /dev/null
            OUTPUT=$(terraform plan -detailed-exitcode -no-color 2>&1)
            EXIT_CODE=$?

            if [ $EXIT_CODE -eq 2 ]; then
              DRIFT_FOUND=true
              REPORT="${REPORT}\n### Drift in ${dir}\n\`\`\`\n${OUTPUT}\n\`\`\`\n"
            elif [ $EXIT_CODE -ne 0 ]; then
              REPORT="${REPORT}\n### Error in ${dir}\n\`\`\`\n${OUTPUT}\n\`\`\`\n"
            fi
            cd -
          done

          echo "drift_found=${DRIFT_FOUND}" >> $GITHUB_OUTPUT
          echo "report<<EOF" >> $GITHUB_OUTPUT
          echo -e "$REPORT" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT

      - name: Open issue if drift detected
        if: steps.plan.outputs.drift_found == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Infrastructure drift detected — ${new Date().toISOString().split('T')[0]}`,
              body: `## Drift Report\n\n${{ steps.plan.outputs.report }}`,
              labels: ['infrastructure-drift']
            })

terraform plan -detailed-exitcode returns exit code 0 (no changes), 1 (error), or 2 (changes detected). Exit code 2 indicates drift. The workflow opens a GitHub issue with the plan output on drift detection.

Responding to drift

Drift has two correct responses:

Apply to converge: the Terraform configuration is the desired state and the manual change was wrong. Run terraform apply. This restores the resource to the declared state.

Import and codify: the manual change was correct and the configuration needs to catch up. Import the resource’s current state into Terraform, then update the configuration to match. This is the right path when an incident required an emergency change that should be made permanent.

Automated auto-remediation (running terraform apply automatically when drift is detected) is tempting but dangerous. A false positive in the plan or an unrelated planned change can cause an unexpected production apply. Alert on drift; remediate with human review.

Refactoring Without Destroying: moved Blocks

The largest operational risk in Terraform refactoring is the rename problem. Renaming a resource in Terraform means the old resource is planned for deletion and the new one for creation. For a stateful resource like an RDS instance, that is a production outage.

The moved block, introduced in Terraform 1.1, solves this by declaring that a resource address has changed without the underlying infrastructure changing:

1
2
3
4
5


# You renamed aws_instance.web to aws_instance.app_server
moved {
  from = aws_instance.web
  to   = aws_instance.app_server
}

After adding the moved block, terraform plan shows:

  # aws_instance.web has moved to aws_instance.app_server

No destroy, no create — just a state rename. Run terraform apply and the state is updated. The block can be removed after all consumers of the module have updated.

Extracting resources into a module

The moved block handles module extraction too:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Resources that were in the root module are now inside module.database
moved {
  from = aws_db_instance.postgres
  to   = module.database.aws_db_instance.postgres
}

moved {
  from = aws_db_subnet_group.postgres
  to   = module.database.aws_db_subnet_group.postgres
}

Converting count to for_each

A common refactor is converting a count-indexed resource to for_each so individual instances can be targeted without affecting the others:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Before: count
resource "aws_iam_user" "deployer" {
  count = 2
  name  = "deployer-${count.index}"
}

# After: for_each
resource "aws_iam_user" "deployer" {
  for_each = toset(["deployer-0", "deployer-1"])
  name     = each.key
}

# Declare the moves
moved {
  from = aws_iam_user.deployer[0]
  to   = aws_iam_user.deployer["deployer-0"]
}

moved {
  from = aws_iam_user.deployer[1]
  to   = aws_iam_user.deployer["deployer-1"]
}

For large-scale refactors across many resources, tfautomv inspects the plan, detects create/delete pairs for identical resources, and generates moved blocks automatically. This handles the case where you are extracting 20 resources into a module and do not want to write 20 moved blocks by hand.

Importing Existing Resources

terraform import brings an existing cloud resource under Terraform management without recreating it. The classic use case is adopting manually created infrastructure.

Since Terraform 1.5, the preferred approach is the import block (over the CLI command), because it is declarative, reviewable in a pull request, and runs as part of the normal plan/apply cycle:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# import.tf
import {
  to = aws_security_group.app
  id = "sg-0abc123def456"
}

import {
  to = aws_vpc.main
  id = "vpc-0abc123def456"
}

Run terraform plan and Terraform shows the import alongside any other planned changes. Run terraform apply and the resources are imported into state. After a successful import, remove the import blocks — they are a one-time migration declaration.

If you are importing an existing resource without a corresponding configuration block, Terraform 1.5+ can generate the configuration from the live resource:

1

terraform plan -generate-config-out=generated.tf

This writes a configuration block for the imported resource to generated.tf. It is a starting point, not production-ready code — review and clean it up before committing.

State Surgery

Sometimes the plan and apply cycle is not enough. State files get corrupted, resources get orphaned, or a manual terraform destroy leaves state entries pointing at things that no longer exist. State surgery is the last resort.

Inspecting state

1
2
3
4
5
6
7
8


# List all resources in state
terraform state list

# Show a specific resource's state
terraform state show aws_instance.app_server

# Show with full detail
terraform state show -json aws_instance.app_server | jq .

Moving resources between state files

When splitting a monolithic stack into smaller stacks, resources need to move from one state file to another. tfmigrate handles this as a declarative migration with dry-run support:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# migration.hcl
migration "state" "move_database_to_own_stack" {
  from_dir = "infrastructure/monolith"
  to_dir   = "infrastructure/database"

  actions = [
    "mv aws_db_instance.postgres aws_db_instance.postgres",
    "mv aws_db_subnet_group.postgres aws_db_subnet_group.postgres",
    "mv aws_security_group.rds aws_security_group.rds",
  ]
}

1
2


tfmigrate plan migration.hcl    # dry run
tfmigrate apply migration.hcl   # execute

Removing orphaned state entries

If a resource was deleted outside Terraform and the state entry is stale:

1
2
3
4
5


# Remove from state without destroying (already gone)
terraform state rm aws_instance.orphaned

# Remove a module's entire state subtree
terraform state rm module.old_database

Replacing a corrupted resource

terraform apply -replace forces a resource to be destroyed and recreated even if the plan shows no changes. Use it when a resource’s live state is corrupt but the state file is clean:

1

terraform apply -replace="aws_instance.app_server"

Pulling and pushing state directly

In emergencies, you can download state, edit it by hand, and push it back. This is dangerous and should be a last resort:

1
2
3
4
5
6
7


# Download current state
terraform state pull > emergency-backup.tfstate

# Edit emergency-backup.tfstate in a text editor

# Push modified state (increments the serial number automatically)
terraform state push emergency-backup.tfstate

Always take a backup with state pull before any state surgery. S3 versioning gives you a fallback, but a local backup is faster to work with.

CI/CD Pipeline Pattern

The standard workflow: plan on PR open/update, apply on merge to main.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


# .github/workflows/terraform.yml
name: Terraform

on:
  push:
    branches: [main]
    paths: ["environments/**", "modules/**"]
  pull_request:
    paths: ["environments/**", "modules/**"]

env:
  TF_VERSION: "1.9.x"
  WORKING_DIR: "environments/prod/networking/vpc"

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: "${{ env.TF_VERSION }}" }

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.TERRAFORM_CI_ROLE_ARN }}
          aws-region: us-east-1

      - working-directory: ${{ env.WORKING_DIR }}
        run: terraform init -input=false

      - working-directory: ${{ env.WORKING_DIR }}
        id: plan
        run: |
          terraform plan -no-color -out=tfplan 2>&1 | tee plan.txt
          echo "plan_output<<EOF" >> $GITHUB_OUTPUT
          cat plan.txt >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT

      - uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## Terraform Plan\n\`\`\`\n${{ steps.plan.outputs.plan_output }}\n\`\`\``
            })

  apply:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: production   # requires approval in GitHub Environments settings
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: "${{ env.TF_VERSION }}" }

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.TERRAFORM_CI_ROLE_ARN }}
          aws-region: us-east-1

      - working-directory: ${{ env.WORKING_DIR }}
        run: terraform init -input=false

      - working-directory: ${{ env.WORKING_DIR }}
        run: terraform apply -auto-approve -input=false

The GitHub Environment (environment: production) enables required reviewers — a human must approve the apply job before it runs. This is the gate that prevents unreviewed plans from reaching production.

For Terragrunt-based repos, terragrunt run-all plan and terragrunt run-all apply replace the single-stack invocations. Gate apply jobs behind environment protection and require the plan output to be reviewed before approving.

OpenTofu

OpenTofu is the CNCF-hosted open-source fork of Terraform created after HashiCorp changed Terraform’s license to BSL in 2023. It is maintained by a community of contributors and major cloud vendors, tracks feature parity with Terraform, and is a drop-in replacement for most workloads.

As of 2025, OpenTofu is at version 1.9.x. It has introduced features not yet in Terraform (provider-defined functions, early variable validation), and both projects continue to diverge gradually. The choice between them is primarily organizational: if your team is comfortable with the Terraform ecosystem and HCP Terraform for remote runs, stick with Terraform. If you have concerns about license constraints on internal tooling, or want to contribute to or depend on a project without vendor control, OpenTofu is the answer.

The migration path is straightforward:

1
2
3
4
5
6
7


# Replace the terraform binary with tofu
brew install opentofu   # or download from github.com/opentofu/opentofu

# Existing state files and configurations are compatible
tofu init
tofu plan
tofu apply

Terragrunt supports both via the --terraform-bin tofu flag.

Honest Trade-offs

What Terraform gets right. The declarative model for cloud infrastructure is correct. Describing what should exist rather than how to create it leads to idempotent, reviewable, auditable changes. The plan/apply cycle — see the diff before applying — is a genuine safety improvement over manual console changes. The provider ecosystem is comprehensive; if a cloud resource exists, there is almost certainly a Terraform resource for it.

The state file is both the best and worst part of Terraform. It is what makes the plan accurate — Terraform knows what it created and can compute the delta. It is also a single point of failure that can be corrupted, lost, or split from reality. Every operational complexity in this post (remote backends, locking, drift detection, state surgery) exists because of the state file. Newer tools like Pulumi and AWS CDK use cloud APIs directly and avoid local state, but they trade the state problem for others. There is no free lunch in infrastructure tooling.

Scale requires discipline, not just tooling. Terragrunt reduces boilerplate; it does not enforce consistency. Module versioning prevents silent breaks; it does not prevent engineers from using source = "../../../modules/vpc" in production. Drift detection alerts; it does not remediate. The tooling provides the capability; the team practices provide the enforcement. A Terraform codebase at scale is primarily a social coordination problem.

The drift problem is chronic, not acute. Every mature Terraform deployment has some drift somewhere. The goal is not zero drift but fast detection and a clear remediation process. Teams that run scheduled drift detection and treat drift issues as engineering work (not incidents) manage it. Teams that only discover drift when a plan fails in CI are perpetually surprised.

HCP Terraform (formerly Terraform Cloud) solves the operational parts. Remote runs, state management, drift detection, OIDC federation, cost estimation, and policy enforcement are all available in HCP Terraform. The free tier covers small teams. If you are building the CI pipeline, backend, and drift detection described in this post and do not enjoy infrastructure-for-your-infrastructure work, HCP Terraform eliminates most of it. The tradeoff is cost ($20/user/month for teams) and a dependency on HashiCorp’s hosted service.