Skip to content
build-3000
devops

Fault Tolerance and Disaster Recovery

RTO/RPO targets, backups you have actually restored, and failover rehearsed until boring.

Before this: Replication and Sharding

Objective

Close the level by planning for bad days: choose RTO/RPO targets honestly, automate encrypted offsite backups with retention, rehearse full restores and primary-loss failover, and hand your runbook to a stranger to execute blind.

Vocabulary

The words to use when directing your agent.

RTO
Recovery time objective - maximum tolerable minutes from disaster to restored service.
RPO
Recovery point objective - maximum tolerable data loss measured in time.
backup retention
How many generations of backups are kept, and for how long.
failover
Switching service to a standby when the primary dies.
restore drill
Actually restoring a backup somewhere real and verifying it - regularly.

Understand it

Backup architecture worth having

Automated on a schedule, encrypted at rest, copied OFFSITE (different region/provider - a backup on the same machine as the data shares its disasters), retention tiered (recent dailies, older weeklies/monthlies), and restoration REHEARSED: pull a random backup, restore to a clean environment, verify row counts and spot-check integrity, time the whole thing. That timing IS your real RTO estimate.

Failure domains and the rehearsal habit

Assume one domain disappears completely - a region, a provider, a disk array. Tolerance means: state replicated or restorable OUTSIDE that domain, DNS/routing able to point somewhere else, and humans who have DONE it before under pressure. Rehearsal converts disaster from improvisation theater into a checklist you have run quarterly.

Core concepts

An unrestored backup is a rumor

Backups earn zero trust until one has been restored, end to end, and the restored system verified against counts and checksums. Corruption propagates too - restoring yesterday's corrupted dump restores the corruption. Hence: automated encrypted backups, offsite copies, tested retention windows, and restore drills scheduled like payroll.

Targets are chosen, then rehearsed

RTO and RPO are business decisions wearing engineering clothes: how much data may we lose (RPO sets backup frequency), how long may we stay down (RTO sets failover machinery). Choosing them costs nothing; MEETING them costs automation and rehearsals. The gap between the two is where disasters live.

Examples

The restore that saved Tuesday

Bad migration corrupts the expenses table Friday night. Restore drill muscle memory kicks in: latest verified nightly pulled, restored to standby in the practiced sequence, counts verified, DNS flipped. Users missed an hour; the team missed none of their practice. The drill WAS the disaster plan.

Same-shelf backups fail together

A startup backed up diligently - onto an external drive sitting beside the server. The office flood took both. Offsite is not paranoia; it is the difference between an outage story and a company-ending one.

Common mistakes

Common mistake

Backups landing on the same disk, machine or region as the data they protect.

Fix

Offsite, different provider or region, encrypted. A backup sharing fate with its data is decoration.

Common mistake

Retention configured once and forgotten until the one backup you need expired.

Fix

Tiered retention reviewed periodically; restore drills PULL FROM REAL ARCHIVE GENERATIONS, not just last night's copy.

Working with your AI agent

Have your agent script the ENTIRE lifecycle: backup job, encryption, offsite copy, verification checksums, and a restore runbook written for a tired stranger at 3am. Then demand adversarial review OF THE RUNBOOK - your agent should try to execute it finding every ambiguity, because ambiguity at 3am becomes downtime.

Build it

The module’s real project. Your agent builds; you direct and inspect.

Project: Phoenix Drill

Give the Ledger its resurrection protocol: scheduled encrypted offsite backups with tiered retention, a stranger-proof restore runbook, a full restore executed and timed, and a primary-loss failover rehearsed against your stated RTO/RPO.

Requirements

  • Automated scheduled backups: encrypted at rest, copied offsite, checksummed
  • Tiered retention configured and DOCUMENTED: dailies, weeklies, monthlies with lifetimes
  • Restore runbook written for an operator with zero context - every command, every verification step
  • Full restore DRILL executed to a clean environment: counts/checksums verified, wall-clock time recorded as your real RTO evidence
  • Primary-loss failover rehearsed: promote standby (or documented equivalent), redirect, verify, fall back
  • RTO/RPO targets stated upfront; drill results compared against them honestly - gaps listed

Context for your agent

Build the Phoenix Drill for my Expense Ledger. Script automated encrypted backups with checksums, offsite copy, and tiered retention - configuration documented. Write the restore runbook for a context-free operator: exact commands, expected outputs, verification checkpoints with counts and checksums. Then coach my drills: (1) full blind restore to a clean environment, timed, (2) primary-loss failover and return. We state RTO/RPO targets FIRST, then grade our results against them honestly, gaps and all.

Verify it - twice

Check every item yourself first. Then hand your agent the prompt and compare verdicts.

Self-verified0/10
Functional
Architecture
Security
Testing
Production

Verification prompt - paste into your AI agent

Execute my Phoenix Drill's doom rehearsal as the 3am operator who has never seen this system - judge only the runbooks and artifacts, not my memories. (1) Perform the blind restore EXACTLY per runbook: note every ambiguity, stall and unstated prerequisite, (2) verify the restored Ledger against counts and checksums - would you trust it with real money data tonight?, (3) delete the newest backup and restore from an OLDER tier - does retention behave as documented?, (4) grade my RTO/RPO claims against the timings you just produced. Finish with the runbook's weakest sentence - the one that would cost us most at 3am.

Run this against “Fault Tolerance and Disaster Recovery” after your own checklist pass. Where the agent and you disagree, investigate before moving on.

Review questions

Answer out loud. If you cannot, revisit before moving on.

  1. State your RTO and RPO - and what the drill proved versus what you hope.

  2. Why does a restore drill pull from OLD generations, not last night?

  3. What failure domain does your offsite copy NOT share with production?

You can now

  • Choose and defend RTO/RPO targets as business-informed engineering
  • Operate encrypted, offsite, retained backups that restore for real
  • Rehearse disaster until recovery is a checklist, not a crisis

Up next

LLM Fundamentals