Skip to content
build-3000
backend

Distributed Systems Thinking

Latency, partial failure and consistency tradeoffs as the default condition - not the exception.

Before this: Failure Handling

Objective

Reason about systems spanning multiple processes and machines: why network calls are not function calls, what partial failure means for correctness, and how to choose - and document - consistency and latency tradeoffs deliberately.

Vocabulary

The words to use when directing your agent.

distributed system
Software running across multiple processes or machines that coordinate over a network.
partial failure
Some components working while others fail - the state distributed systems live in.
latency
The time an operation takes end to end; across networks it dominates local computation.
consistency
The guarantee that all readers see the same data at the same moment.
partition
A break in the network splitting parts of a system from each other.

Understand it

The questions every boundary asks

Split any work across two processes and inherit four questions: What if the other side is slow? (timeouts - Level 3). What if the call never arrives? (retries with dedup). What if both sides think they succeeded? (reconciliation). Who is right when they disagree? (a documented decision). Systems that never answered these on paper answer them at 3am by accident.

Start with two, not twenty

You learn distributed thinking by splitting ONE responsibility out of your app - say, report generation becoming its own service reading from the same store - and observing everything that gets harder: deployment, monitoring, failure isolation, version skew between halves. Two services already exhibit nearly every disease of two hundred; the scale changes, the physics does not.

Core concepts

A network call is not a function call

Calling a function in-process either runs or throws. Calling another machine can run, throw, hang, vanish mid-flight after doing the work, or succeed while the reply is lost forever. That extra universe of outcomes - especially 'did it actually happen?' - is the entire subject of distributed systems.

Tradeoffs are chosen, not escaped

When a partition strikes, you choose between refusing requests (staying consistent) and serving possibly-stale answers (staying available). Neither is free, and the right choice varies per feature: bank balances lean consistent, product pages lean available. Seniority here is writing down which you chose, where, and why.

Examples

The vanishing write

Ledger calls the reporting service to log a summary update. The service receives it, writes it, then crashes before responding. Ledger times out and retries. Was the update applied once, twice, or never? Without idempotent handling nobody knows - this exact fog motivates the next modules.

Two services, one user question

'Did my expense save?' now has a compound answer: the API committed it, but the search index lags thirty seconds behind. The honest UI says 'saved - indexing shortly'. Distributed literacy is knowing WHICH truth each component holds.

Common mistakes

Common mistake

Designing as if calls between services behave like local functions.

Fix

For every cross-service call ask: what if it hangs, duplicates, or succeeds invisibly? Write the answer down before coding against it.

Common mistake

Debugging cross-service issues without shared correlation ids.

Fix

Propagate request ids across every hop (Level 3's Watchtower habit) BEFORE splitting anything further.

Working with your AI agent

Before any split, make your agent produce a boundary contract: what crosses the line, what happens on timeout, duplicate delivery, and reorder - plus which consistency choice each read makes. Reviewing that one page prevents entire categories of accidental architecture.

Build it

The module’s real project. Your agent builds; you direct and inspect.

Project: Split Decision

Extract ONE responsibility of the Ledger into a second local service, put a fault-injecting proxy between them, and write the decision memo documenting every tradeoff you chose - then attack your own design with chaos probes.

Requirements

  • One capability extracted into a separate process communicating over HTTP - reports or notifications are natural candidates
  • Fault-injection proxy between services: configurable latency, dropped responses, duplicate responses
  • Boundary contract written: endpoints, timeout values, retry policy, duplicate-handling expectations
  • Consistency decisions documented per feature: what may lag, what must not, and what users see during lag
  • Chaos session performed: kill the second service mid-request, observe behavior, record findings against the memo

Context for your agent

Help me extract one Ledger capability into a second local service with a fault-injecting proxy between us. Before code: draft the boundary contract together - who calls whom, timeouts, retry rules, duplicate-delivery assumptions, and my consistency choices per feature. Then implement minimally. Then coach my chaos drills: inject latency, drop responses, duplicate them, kill the service mid-call - I record observed behavior against our memo and we patch the gaps honestly.

Verify it - twice

Check every item yourself first. Then hand your agent the prompt and compare verdicts.

Self-verified0/8
Functional
Architecture
Security
Testing
Production

Verification prompt - paste into your AI agent

Interrogate my Split Decision like a systems architect auditing a boundary drawn in pencil. Verify PASS or FAIL with evidence: (1) run my chaos drills yourself through the fault proxy - do behaviors match what the memo claims?, (2) find the undefined case: name ONE failure mode my boundary contract does not cover, (3) check correlation ids actually cross the boundary in logs, (4) challenge my consistency choices: pick one feature and argue the opposite tradeoff. Finish by grading the memo itself - would you let a team ship based on it?

Run this against “Distributed Systems Thinking” after your own checklist pass. Where the agent and you disagree, investigate before moving on.

Review questions

Answer out loud. If you cannot, revisit before moving on.

  1. Which outcomes differ between calling a function and calling another machine?

  2. State one consistency-versus-availability decision YOU made, and its cost when wrong.

  3. What did killing the second service reveal that happy-path testing never could?

You can now

  • Reason about latency, duplication and partial failure at boundaries
  • Write boundary contracts and consistency decisions down
  • Chaos-test designs before production does it for free

Up next

Event-Driven Architecture