Fault Tolerance and Disaster Recovery
RTO/RPO targets, backups you have actually restored, and failover rehearsed until boring.
Before this: Replication and Sharding
Objective
Close the level by planning for bad days: choose RTO/RPO targets honestly, automate encrypted offsite backups with retention, rehearse full restores and primary-loss failover, and hand your runbook to a stranger to execute blind.
Vocabulary
The words to use when directing your agent.
- RTO
- Recovery time objective - maximum tolerable minutes from disaster to restored service.
- RPO
- Recovery point objective - maximum tolerable data loss measured in time.
- backup retention
- How many generations of backups are kept, and for how long.
- failover
- Switching service to a standby when the primary dies.
- restore drill
- Actually restoring a backup somewhere real and verifying it - regularly.
Understand it
Backup architecture worth having
Automated on a schedule, encrypted at rest, copied OFFSITE (different region/provider - a backup on the same machine as the data shares its disasters), retention tiered (recent dailies, older weeklies/monthlies), and restoration REHEARSED: pull a random backup, restore to a clean environment, verify row counts and spot-check integrity, time the whole thing. That timing IS your real RTO estimate.
Failure domains and the rehearsal habit
Assume one domain disappears completely - a region, a provider, a disk array. Tolerance means: state replicated or restorable OUTSIDE that domain, DNS/routing able to point somewhere else, and humans who have DONE it before under pressure. Rehearsal converts disaster from improvisation theater into a checklist you have run quarterly.
Core concepts
An unrestored backup is a rumor
Backups earn zero trust until one has been restored, end to end, and the restored system verified against counts and checksums. Corruption propagates too - restoring yesterday's corrupted dump restores the corruption. Hence: automated encrypted backups, offsite copies, tested retention windows, and restore drills scheduled like payroll.
Targets are chosen, then rehearsed
RTO and RPO are business decisions wearing engineering clothes: how much data may we lose (RPO sets backup frequency), how long may we stay down (RTO sets failover machinery). Choosing them costs nothing; MEETING them costs automation and rehearsals. The gap between the two is where disasters live.
Examples
The restore that saved Tuesday
Bad migration corrupts the expenses table Friday night. Restore drill muscle memory kicks in: latest verified nightly pulled, restored to standby in the practiced sequence, counts verified, DNS flipped. Users missed an hour; the team missed none of their practice. The drill WAS the disaster plan.
Same-shelf backups fail together
A startup backed up diligently - onto an external drive sitting beside the server. The office flood took both. Offsite is not paranoia; it is the difference between an outage story and a company-ending one.
Common mistakes
Common mistake
Backups landing on the same disk, machine or region as the data they protect.
Fix
Offsite, different provider or region, encrypted. A backup sharing fate with its data is decoration.
Common mistake
Retention configured once and forgotten until the one backup you need expired.
Fix
Tiered retention reviewed periodically; restore drills PULL FROM REAL ARCHIVE GENERATIONS, not just last night's copy.
Working with your AI agent
Have your agent script the ENTIRE lifecycle: backup job, encryption, offsite copy, verification checksums, and a restore runbook written for a tired stranger at 3am. Then demand adversarial review OF THE RUNBOOK - your agent should try to execute it finding every ambiguity, because ambiguity at 3am becomes downtime.
Build it
The module’s real project. Your agent builds; you direct and inspect.
Project: Phoenix Drill
Give the Ledger its resurrection protocol: scheduled encrypted offsite backups with tiered retention, a stranger-proof restore runbook, a full restore executed and timed, and a primary-loss failover rehearsed against your stated RTO/RPO.
Requirements
- Automated scheduled backups: encrypted at rest, copied offsite, checksummed
- Tiered retention configured and DOCUMENTED: dailies, weeklies, monthlies with lifetimes
- Restore runbook written for an operator with zero context - every command, every verification step
- Full restore DRILL executed to a clean environment: counts/checksums verified, wall-clock time recorded as your real RTO evidence
- Primary-loss failover rehearsed: promote standby (or documented equivalent), redirect, verify, fall back
- RTO/RPO targets stated upfront; drill results compared against them honestly - gaps listed
Context for your agent
Build the Phoenix Drill for my Expense Ledger. Script automated encrypted backups with checksums, offsite copy, and tiered retention - configuration documented. Write the restore runbook for a context-free operator: exact commands, expected outputs, verification checkpoints with counts and checksums. Then coach my drills: (1) full blind restore to a clean environment, timed, (2) primary-loss failover and return. We state RTO/RPO targets FIRST, then grade our results against them honestly, gaps and all.
Verify it - twice
Check every item yourself first. Then hand your agent the prompt and compare verdicts.
Verification prompt - paste into your AI agent
Execute my Phoenix Drill's doom rehearsal as the 3am operator who has never seen this system - judge only the runbooks and artifacts, not my memories. (1) Perform the blind restore EXACTLY per runbook: note every ambiguity, stall and unstated prerequisite, (2) verify the restored Ledger against counts and checksums - would you trust it with real money data tonight?, (3) delete the newest backup and restore from an OLDER tier - does retention behave as documented?, (4) grade my RTO/RPO claims against the timings you just produced. Finish with the runbook's weakest sentence - the one that would cost us most at 3am.
Run this against “Fault Tolerance and Disaster Recovery” after your own checklist pass. Where the agent and you disagree, investigate before moving on.
Review questions
Answer out loud. If you cannot, revisit before moving on.
State your RTO and RPO - and what the drill proved versus what you hope.
Why does a restore drill pull from OLD generations, not last night?
What failure domain does your offsite copy NOT share with production?
You can now
- Choose and defend RTO/RPO targets as business-informed engineering
- Operate encrypted, offsite, retained backups that restore for real
- Rehearse disaster until recovery is a checklist, not a crisis
Up next
LLM Fundamentals