Terraform at Scale: What Changes After 200 Resources

Below roughly 50 resources, almost any Terraform structure works. One state file, a handful of loosely organized modules, applies run by whoever's making the change: none of it matters yet because the blast radius of a mistake is small and the team is probably one or two people who both know the whole system.

Past 200 resources, the decisions made early, one state file or many, module boundaries drawn around teams or around services, start to determine how fast the team can ship and how long an outage lasts when something goes wrong. This is what changed on the way past that line, including a four-hour recovery that traced back to a decision made when the codebase was a tenth of its current size.

Key takeaways

• A shared state file with no locking discipline let two pipelines apply concurrently. The result was a plan that looked clean but silently dropped a security group reference, not a corrupted file.
• State locking exists specifically to prevent this: HashiCorp's own guidance is that remote backends should use locking "to prevent concurrent runs of Terraform against the same state."
• The fix was smaller blast radius, not a bigger tool: state split per service boundary, mandatory remote locking, and a second-reviewer policy for any apply touching networking or IAM.
• Design modules for the reviewer, not the author. A module that's clever to write but opaque to review is a liability the moment its author isn't in the room.

The four-hour incident

A shared state file, used by two separate deploy pipelines, had no locking discipline enforced. Both pipelines applied within the same window. Terraform's state locking exists precisely to prevent this: as HashiCorp's documentation puts it, "for fully-featured remote backends, Terraform can also use state locking to prevent concurrent runs of Terraform against the same state" (Terraform Remote State documentation). Without it enforced, both applies proceeded, and the second one's plan was computed against a state snapshot that was already stale.

The result wasn't corruption in the traditional sense, the state file itself stayed syntactically valid. It was worse: a plan that looked entirely clean but silently dropped a security group reference that the other pipeline had just added. Recovery took longer than the outage itself, because the state history didn't clearly show when the drift was introduced. Reconstructing the timeline meant diffing state snapshots against deploy logs from both pipelines to find the exact apply where the reference disappeared.

The fix: smaller blast radius, not a bigger tool

The instinct after an incident like this is to reach for more tooling, a policy engine, a fancier CI gate, a state-diffing bot. What actually fixed it was structural:

• Split state per service boundary. One state file per service means a bad apply in one service can't silently corrupt another service's resources. It also shrinks the blast radius of the locking gap itself: two pipelines can still race, but only within one service's resources, not across the whole account.
• Mandatory remote locking, enforced, not assumed. The backend already supported locking. The gap was that nothing enforced its use consistently across both pipelines. Enforcement, not availability, was the actual fix.
• A second-reviewer policy for networking and IAM changes. Any apply touching security groups, IAM policies, or network ACLs requires a second engineer to review a fresh plan before it runs. This is deliberately narrow: it's not a review-everything policy, which would just slow down routine changes without addressing where the actual risk concentrates.

Module design: optimize for the reviewer

The other lesson from scaling past 200 resources is about module boundaries, and it's less about Terraform mechanics than about who has to read the diff six months from now. HashiCorp's own guidance on module composition recommends a flat hierarchy over deeply nested modules, with dependencies passed in from a parent rather than each module fetching its own: "we call this flat style of module usage module composition, because it takes multiple composable building-block modules and assembles them together to produce a larger system" (HashiCorp Terraform Module Composition documentation).

That composition pattern matters here for a specific reason: a module that's clever to write but opaque to review becomes a liability the moment the original author isn't in the room, and past 200 resources, the original author is eventually not going to be in the room for every change. A module boundary drawn around a service (this module is "the payments database and its IAM role") is reviewable by anyone on the team who understands that service. A module boundary drawn around a Terraform abstraction (this module is "generic RDS instance with 40 configurable variables") is only reviewable by whoever wrote it or has memorized its variable surface.

What to check before you hit 200 resources

If you're earlier in this curve, three questions are worth answering before growth forces the answer on you:

• Is state locking actually enforced in every pipeline that can run an apply, or just supported by the backend?
• Can you trace, from state history alone, exactly which apply introduced any given resource or attribute? If not, your recovery time after an incident will look like the four hours above.
• Are your module boundaries drawn around services a new team member could name, or around Terraform patterns only the original author fully understands?

Frequently asked questions

Does splitting state per service slow down cross-service changes? It adds friction for changes that genuinely span services, since you need to sequence applies or use remote state data sources to pass values between them. That friction is the point: cross-service changes should get more scrutiny than single-service ones, not less.

Is HCP Terraform / Terraform Cloud a substitute for these fixes? It solves the locking-enforcement problem well since remote runs are serialized by default, but it doesn't solve module boundary design or review policy on its own. Those are decisions your team makes regardless of which backend you run.

How do you decide where a service boundary actually is? Draw it around what a single team owns and can reason about end to end, not around a natural-seeming AWS resource grouping. If two teams routinely need to review each other's changes in the same state file, the boundary is probably in the wrong place.