The plan wanted to replace the production database

The client's AWS account had been built by hand over a couple of years. Names invented per resource, security groups nobody could justify, leftovers that existed because someone once needed them on a Tuesday. Two accounts, a dev environment that had stopped working, secrets in .env files, and long-lived IAM access keys doing the authentication.

Our job was the boring version of a rescue: modular Terraform underneath, Terragrunt on top, three environments that actually resemble each other, and a pipeline that can tell you what it is about to do.

The part I keep retelling isn't the architecture. It's the afternoon a routine refactor put a production database one apply away from being replaced.

The change that looked safe

The database had been created early, by a community module, while the environment was still being stood up. Later we replaced that module with our own wrapper. Same engine, same instance class, same data, different code path.

Terraform does not track a resource by what it is. It tracks it by address: module.<name>.<type>.<name>. Change the module and you have changed the identity of the thing, as far as state is concerned. Everything that follows is the tool being consistent.

The plan itself is unremarkable to look at, which is exactly the danger. Two lines carry the whole story, and they sit in the middle of a wall of attribute diffs:

# module.db.aws_db_instance.this will be destroyed
# module.rds.aws_db_instance.this will be created

Plan: 1 to add, 0 to change, 1 to destroy.

1 to destroy on an environment where the numbers are usually zero is the line that should stop your hand. The fix is not clever. It is one command, run before the apply:

terraform state mv \
  module.db.aws_db_instance.this \
  module.rds.aws_db_instance.this

State now points the new address at the existing instance, and the next plan has nothing to say. Thirty seconds of work, on the correct side of an outage.

A migration rarely kills you with destroy. It kills you with a resource address.

Why we were rewriting the module at all

The reason is worth telling, because it is the more common bug.

The RDS master password had been generated by AWS, and it started with a character the migration tool read as the beginning of a comment. The quick workaround at the time was to pass a custom password as an environment variable during terraform apply. That unblocks the afternoon and quietly buys you drift: the value lives outside the configuration, so every subsequent plan has an opinion about it.

The wrapper exists to take the password out of that path entirely:

ephemeral "random_password" "master" {
  length           = 32
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "aws_secretsmanager_secret_version" "master" {
  secret_id                = aws_secretsmanager_secret.master.id
  secret_string_wo         = ephemeral.random_password.master.result
  secret_string_wo_version = 1
}

resource "aws_db_instance" "this" {
  # ...
  manage_master_user_password = false
}

secret_string_wo is a write-only argument, supported since Terraform 1.11. The provider accepts the value and state never records it. After bootstrap, password rotation lives outside Terraform, in Secrets Manager, which is where it belonged in the first place.

That is the trade the wrapper makes, and it is worth saying plainly: you gain a state file with no password in it, and you accept that rotating the password is now a Secrets Manager operation with a manual step, not a terraform apply.

The DRY trap that came first

Before any of that, we spent a stretch of the project making the repository worse.

The instinct is right. Units across three environments repeat a lot of the same values, so you put the shared ones in a single defaults.hcl and read it everywhere. The mistake is the block you put them in.

In practice the first shape looks like this, in every single unit:

locals {
  defaults = read_terragrunt_config(find_in_parent_folders("defaults.hcl"))
}

inputs = {
  name        = local.defaults.locals.name
  environment = local.defaults.locals.environment
  tags        = local.defaults.locals.tags
  # and on, and on, for every value the module takes
}

Nothing here is shared. The location of the values is shared; the wiring is copied. One of those config files reached about 700 lines before we accepted that the pattern, not the file, was the problem.

The second shape moves the same values into inputs on the parent and lets the merge do the work:

# root.hcl
inputs = {
  environment = "prod"
  tags = {
    owner   = "platform"
    managed = "terragrunt"
  }
}

# unit
include "root" {
  path           = find_in_parent_folders("root.hcl")
  merge_strategy = "deep"
}

inputs = {
  instance_class = "db.r6g.large"
}

The catch nobody documents loudly enough

Deep merge is not uniform across types, and the difference will find you through tags:

Tags as a map are fine, and that is the shape to prefer. Tags as a list are not: an environment can append to the parent's list, but it cannot replace an entry in it. Where we needed genuinely different list values per environment, we gave the keys different names and let the module recombine them, which is uglier than it sounds in a sentence and less ugly than a list you cannot override.

remote_state and generate blocks do not deep merge at all, which is its own small surprise the first time you rely on it.

What the pipeline learned

The last piece was CI. The first implementation ran a matrix job over the units, which works and scales badly: the matrix is a hand-maintained copy of the dependency graph, and it drifts from the repository the moment someone adds a unit.

Terragrunt already knows the graph, and since the 1.x filter syntax it will also work out what a commit touched:

terragrunt run --all --filter '[main...HEAD]' -- plan

Two related habits came out of the same review, both cheap:

• Run terragrunt validate across the whole codebase, not just the modules, so inputs and references between units are checked too
• Build the dependency cache once, during validate, and reuse it for plan and apply instead of re-downloading on every step

And one vocabulary fix that saves confusion in every later conversation: in Terragrunt, a folder with a config in it is a unit. A stack is a generated tree of units. Calling everything a stack made half our documentation ambiguous.

What I would tell the version of me from March

The database was never actually lost, so the honest version of this story is not heroic. Someone read the plan properly. That is the entire safety mechanism, and writing it out like that is uncomfortable.

So the process changes that came out of it are dull on purpose. Any plan touching a stateful resource gets read for identity first, counts second. Module swaps on live resources come with the state mv written into the pull request description, before the apply, where a reviewer can see it. And the DRY refactor waits until the environment is stable, because a config rewrite and a resource-address change in the same week is how you lose track of which one moved the ground.

The migration itself worked. Three environments, deny-all security groups, secrets in Secrets Manager, a pipeline that plans only what changed. None of that is the part I remember.

Has your plan ever wanted to replace something that could not be replaced? I am collecting the ones where the tool was technically right, because those are the interesting failures.