The big idea
A disaster recovery plan is a commitment on paper until teams test it. Progressively more demanding exercises reveal different gaps, from unclear ownership to failures that were not anticipated in the plan.
Why it matters
A planned failover can show that traffic moves and an application works in the recovery region. It cannot, by itself, establish that teams will coordinate under pressure or that recovery capacity is ready when needed. Findings from each exercise should inform the recovery boundary, business targets, and commitments involving dependencies.
Key insights
- 01
Different exercises answer different readiness questions.
A tabletop walkthrough clarifies scope and ownership without changing production. A planned failover tests the mechanism under known conditions. An unannounced drill tests whether responders can execute without preparation, while fault injection explores dependency failures that the plan may not have anticipated.
- 02
Healthy-looking applications can still fail users through a shared dependency.
In the case described, a planned failover found that shared-platform DNS records still pointed to the primary region 45 minutes into the exercise. Downstream applications appeared healthy but could not serve users. Recovery took more than twice the five-hour target, making the measured result important for discussions with the business owner.
- 03
Recovery knowledge concentrated in one person is a readiness risk.
An unannounced drill exposed a database-consistency procedure that only one engineer understood. When that engineer was unavailable, the team spent 90 minutes finding and interpreting the procedure. The drill tested the team’s ability to execute, not just whether recovery was technically possible.
Practical takeaways
- 01
Walk through the recovery plan and assign owners and backups to its steps.
Use the walkthrough to clarify who can declare failover, identify dependencies outside the current scope, and find steps that rely on one person or an unnamed team.
- 02
Prepare the recovery environment and check capacity assumptions before a failover.
The case relied on creating and scaling resources during recovery without first checking regional capacity or account quotas. The source recommends provisioning ahead of time, configuring auto scaling groups, and removing runbook steps that assume resources must be created during the event.
- 03
Measure recovery results against the stated objectives.
During a planned failover, compare each application’s actual recovery time with its recovery time objective. Validate data consistency against the recovery point objective, and record manual steps and dependencies that recover later than expected.