Failure Handling
Timeouts everywhere, circuit breakers, bounded retries and honest degradation when dependencies wobble.
Before this: Observability Foundation
Objective
Design for partial failure as the normal case: explicit timeouts on every remote call, retries bounded with backoff, circuit breakers stopping cascade damage, graceful degradation keeping features alive in reduced form - rehearsed with chaos drills.
Vocabulary
The words to use when directing your agent.
- timeout
- The maximum wait you grant a call before giving up and moving on.
- retry storm
- Aggressive automatic retries amplifying an outage into a self-inflicted flood.
- circuit breaker
- A guard that stops calling a failing dependency temporarily, letting it recover.
- graceful degradation
- Continuing reduced service - cached data, fewer features - instead of hard failure.
- fallback
- The designed alternative answer used when the primary path is unavailable.
Understand it
Timeouts are love letters to your future self
Every outbound call - HTTP fetch, database query, queue operation - carries an explicit maximum wait sized honestly for the operation. Without one, a stalled dependency holds your threads hostage while requests pile up: your outage is now THEIR outage, multiplied. With timeouts, failure arrives fast, small and diagnosable instead of slow, total and mysterious.
Breakers and degradation ladders
Circuit breaker logic: after consecutive failures cross a threshold, STOP calling - trip open. Requests during the open window receive the fallback instantly. Periodically allow one probe through (half-open); success closes the circuit again. Meanwhile features degrade along a designed ladder: fresh data, then cached data flagged 'as of when', then static default, then an honest error. Each rung is chosen in calm daylight, never invented at midnight.
Core concepts
Everything remote eventually fails
Networks partition, services restart, providers deploy maintenance windows mid-your-peak. On a long enough timeline every dependency WILL be down while users watch. Systems assuming otherwise do not avoid failure - they just meet it unrehearsed. The professional posture: partial failure is Tuesday, design for it.
Fail smaller, not harder
One dead dependency should cost you one degraded feature - not a hung server, not cascading collapse. Isolation comes from three boring tools applied everywhere: explicit timeouts (no call hangs forever), bounded retries with backoff (no storm amplification), and circuit breakers (stop feeding a failing dependency requests it cannot answer).
Examples
Currency rates, degraded gracefully
Ledger shows expenses converted to your currency using an external rates API. API dies? Breaker trips within seconds; UI switches to yesterday's cached rates labeled 'rates as of yesterday'; conversions remain possible, slightly stale, honestly labeled. Nobody's evening ruined.
The retry that made it worse
Dependency slows to five-second responses. Naive clients time out at three seconds, retry instantly, doubling load - the struggling service now faces twice the traffic. Bounded retries WITH backoff and jitter refuse to amplify: few attempts, spaced out, capped.
Common mistakes
Common mistake
Relying on default timeouts - or none - because 'it usually responds fast'.
Fix
Every remote call declares its maximum wait explicitly. Usually-fast is not a contract; timeouts are.
Common mistake
Unlimited immediate retries 'until it works'.
Fix
Cap attempts, space them with growing delays and jitter. Otherwise your resilience code becomes a DDoS bot loyal to you.
Working with your AI agent
Demand a dependency inventory: every outbound call listed with its timeout value, retry policy, breaker thresholds and fallback behavior - BEFORE wiring features that use it. Centralize HTTP client defaults so the table is enforced by code, not convention. A dependency without a row in that table is an unaudited risk.
Build it
The module’s real project. Your agent builds; you direct and inspect.
Project: Storm Doors
Harden the Ledger against a wobbling world: add a currency-conversion feature backed by a keyless external rates API, wrapped in explicit timeouts, bounded backoff retries, a visible circuit breaker, stale-cache fallback and a kill switch - then run a chaos drill and document the timeline.
Requirements
- External rates call carries explicit timeout; no other outbound call lacks one either
- Retries bounded: max attempts, exponential-ish backoff with jitter, cap enforced
- Circuit breaker trips after consecutive failures, half-open probes recover - state visible in metrics
- Degradation ladder implemented: fresh rates, cached rates labeled with age, conversion disabled with honest message
- Kill switch env var disables the external feature outright without redeploying code
- Chaos drill documented: network cut mid-flow, behavior timeline recorded minute by minute
Context for your agent
Add currency-conversion display to my Expense Ledger backed by a keyless FX-rates API. Wrap it properly: explicit per-call timeout, bounded retries with jittered backoff, circuit breaker whose state shows in my metrics, fallback ladder serving cached rates with visible age labels, kill-switch env var. Build a dependency-inventory table first covering ALL my outbound calls with timeout/retry/fallback columns - fill gaps found elsewhere too. Then coach my chaos drill: I cut the network, we narrate the timeline together.
Verify it - twice
Check every item yourself first. Then hand your agent the prompt and compare verdicts.
Verification prompt - paste into your AI agent
Run a hostile resilience review of my Storm Doors - you are the hurricane. Attempt each: (1) sever the rates API entirely - walk my degradation ladder rung by rung and confirm labels stay honest, (2) make responses SLOW rather than dead - did timeouts hold or did requests pile up?, (3) watch my retries during an outage - prove they stayed bounded and did not amplify the incident, (4) force the breaker open then partially heal the dependency - did half-open probing behave as documented? Grade every DoD item, then nominate the dependency most likely to ruin my worst day and whether my doors hold against it.
Run this against “Failure Handling” after your own checklist pass. Where the agent and you disagree, investigate before moving on.
Review questions
Answer out loud. If you cannot, revisit before moving on.
Recite your dependency inventory: who talks to the outside world, and with what timeout?
Why do immediate unlimited retries convert outages into self-inflicted wounds?
Describe your degradation ladder for the currency feature - and who chose each rung.
You can now
- Give every remote call explicit timeouts and bounded retries
- Stop cascades with circuit breakers and honest fallbacks
- Rehearse partial failure until it is boring
Up next
Distributed Systems Thinking