Resilience Engineering for SREs: Two Cloud Outages and What They Teach

The goal is never "zero failures." It's bounding the impact of the failures you know are coming, and proving the recovery path works before an incident forces you to use it. The two outages below make that case from opposite directions.

Key takeaways

• Resilience is budgeted, not assumed. SLOs set the target, the error budget sets how much failure you can spend, and RTO/RPO set how fast and how lossy recovery may be.
• The October 20, 2025 US-EAST-1 outage started with a race condition in DynamoDB's automated DNS management that left an endpoint record empty, then cascaded through EC2, NLB, and Lambda for roughly 15 hours (AWS post-event summary).
• The March 2026 AWS Gulf-region outages were physical: drone strikes and a fire the weekend of March 1 knocked two of three availability zones offline in ME-CENTRAL-1 (UAE) and one in ME-SOUTH-1 (Bahrain); a separate me-south-1 event on March 14 took ~84 services down for 7+ hours (The Register).
• Static stability beats reactive failover. Running capacity that survives a dead control plane recovers faster than anything that must make new API calls mid-incident.

DR vs resilience: the SRE reframing

Classic disaster recovery asks "how do we get back after it breaks?" Resilience engineering asks "how do we keep serving while it breaks, and bound the damage?" The two share vocabulary but differ in posture. DR is the recovery floor; resilience is the day-to-day design that keeps most incidents below your users' notice.

The numbers that anchor both:

• SLO — the reliability target you promise (e.g. 99.9% of requests succeed over 28 days).
• Error budget — the inverse of the SLO. It is your permission to fail: spend it on releases and risk, protect it when it runs low.
• RTO (Recovery Time Objective) — maximum acceptable time to restore service.
• RPO (Recovery Point Objective) — maximum acceptable data loss, in time.
• Blast radius — how far a single failure can propagate. Shrinking it is most of the job.

Downtime is the reason these targets exist. In ITIC's 2024 survey of over 1,000 firms, a single hour of downtime exceeded $300,000 for more than 90% of mid-size and large enterprises, and 41% put their hourly cost between $1 million and over $5 million (ITIC, 2024). Now apply that rate to a 15-hour outage — or a multi-day regional loss.

Case study 1: the October 20, 2025 US-EAST-1 cascade (a control-plane failure)

Per AWS's own post-event summary, the trigger was a latent race condition in DynamoDB's automated DNS management. Two independent "enactors" update the DNS record for the regional DynamoDB endpoint. One stalled on high latency; while it lagged, another applied a newer plan. When the slow enactor finally ran — checking state only once — it applied its now-stale plan and overwrote the DNS record with an empty set. All enactors converged on "empty," and nothing backfilled it.

The result: dynamodb.us-east-1.amazonaws.com stopped resolving. Then the dominoes fell, because AWS's own control plane leans on DynamoDB:

• EC2 DropletWorkflow Manager (DWFM) stores state in DynamoDB. Queries timed out, droplet leases expired, hosts were marked unavailable, and the subsystem hit congestive collapse — new EC2 launches failed for hours.
• Network Load Balancers saw fleet-wide health-check failures, driving connection errors.
• Lambda, API Gateway, Cognito, and Route 53 degraded.

Downstream, Snapchat, Reddit, Fortnite, Coinbase, Venmo, Signal, and many more went dark worldwide — from one region's fault. AWS marks the event from 11:48 PM PDT Oct 19 to ~2:20 PM PDT Oct 20, about 15 hours in three overlapping waves.

Case study 2: the March 2026 AWS Gulf-region outages (a physical failure)

Where US-EAST-1 was a logical fault in automation, the Gulf outages were kinetic — the risk SREs most often forget lives under the abstraction. Over the weekend of March 1, 2026, drones struck AWS data-center facilities in the UAE; the impact sparked a fire, sprinklers discharged, and local authorities cut power to the site to contain the blaze (The Register).

The correlated, physical nature of the loss is what makes it an SRE lesson:

• Two of three availability zones went offline in the UAE region (ME-CENTRAL-1), plus one AZ in Bahrain (ME-SOUTH-1). A multi-AZ deployment that assumes zones fail independently does not survive that.
• Customers hit included Emirates NBD, First Abu Dhabi Bank, Abu Dhabi Commercial Bank, ride-hailing platform Careem, and data-cloud firm Snowflake, plus payments platforms Hubpay and Alaan (Hosting Discussion).
• A second event on March 14, 2026 took the Bahrain me-south-1 region down again, knocking roughly 84 services offline for over seven hours.

Multi-AZ is designed to survive the loss of one zone from independent causes — a power feed, a cooling unit, a switch. A single physical event that takes two of three zones at once breaks the independence assumption the whole design rests on. The only real defense at that blast radius is a second region, geographically separated, with data already there.

The four recovery strategies, seen through blast radius

AWS's Well-Architected guidance frames DR as four strategies on a cost-versus-RTO curve. Map each to the blast radius it actually survives:

Strategy	How it works	RTO	Survives	Cost
Backup & Restore	Restore from backups into a new environment	Hours	Region loss (slowly)	$
Pilot Light	Data replicated live; minimal infra kept warm	Tens of minutes	Region loss	$$
Warm Standby	Scaled-down full stack, always running	Minutes	Region loss	$$$
Multi-site Active/Active	Full stack live in multiple regions serving traffic	Near zero	Region loss, transparently	$$$$

Source: AWS Well-Architected — Disaster Recovery of Workloads. Multi-AZ alone appears on none of these rows: it survives a zone, not a region. Both 2025-2026 outages were region-scoped.

SRE lessons: what actually changes your on-call life

1. A region is a single point of failure — a zone even more so. US-EAST-1 defaulted its way into everyone's critical path; the Gulf event proved zones can fail together. Tie your DR strategy to regional loss for anything with a tight SLO.
2. Design for static stability. Both incidents crippled the ability to launch and route, not just stored data. Existing capacity must keep serving when the control plane is dead. Never make serving live traffic depend on making new API calls.
3. Map transitive dependencies, including your provider's. Teams that "didn't use DynamoDB" still fell over because EC2, Lambda, and NLB did. Your real dependency graph includes the cloud's internals.
4. Budget RTO for the cascade drain, not the root-cause fix. Backlogs, retries, and cold caches extend outages long after the trigger is patched.
5. Physical risk is real risk. Fire, power cuts, and now kinetic strikes are in-scope threats. Region selection is a resilience decision, not just a latency one.
6. Untested failover is theater. Run game days and inject failures — region loss, dependency outage, empty DNS — and measure real RTO/RPO against your SLOs.

A practical resilience checklist

• Define SLOs and error budgets per service; let budget burn drive release and risk decisions.
• Assign RTO and RPO to every service and get business sign-off.
• Replicate backups across regions and accounts, and test restores on a schedule — an untested backup is Schrödinger's backup.
• Pick a DR strategy per tier; don't default the whole system to one rung.
• Remove hard single-region dependencies for tight-SLO services; treat multi-AZ as zone-level protection only.
• Design for static stability — running workloads survive a dead control plane.
• Automate failover and DNS cutover; manual steps are slow and error-prone at 3am.
• Run game days / chaos experiments and measure MTTR against targets.
• Keep runbooks current and out-of-band — if your wiki lives on the thing that's down, you can't read it.

Frequently asked questions

Is disaster recovery the same as resilience or high availability?

No. High availability handles routine failures automatically (a dead node, an AZ blip). Disaster recovery is the explicit plan for large, rare events — a whole region, a corrupted database — with a defined RTO/RPO. Resilience engineering is the broader discipline that bounds blast radius so most incidents never reach DR at all.

Would multi-region have saved everyone in these outages?

Only if the failover path didn't itself depend on the failed region, and only if it had been tested. Several "multi-region" setups still failed on Oct 20 because their deployment or DNS tooling lived in US-EAST-1.

Does multi-AZ protect against a regional outage?

No. Multi-AZ survives the loss of one zone from independent causes. The March 2026 Gulf event took two of three zones at once from a single physical cause — exactly the correlated failure multi-AZ does not cover. Only a second region does.

---

Sources

• AWS — Summary of the DynamoDB Service Disruption in US-EAST-1 (official post-event summary)
• ThousandEyes — AWS Outage Analysis: October 20, 2025
• The Register — AWS says drones hit two of its datacenters in UAE
• Hosting Discussion — AWS outage in Gulf reveals the cloud's physical risk
• ITIC — 2024 Hourly Cost of Downtime Report
• AWS Well-Architected — Disaster Recovery of Workloads on AWS