Skip to content
build-3000
backend

Eventual Consistency

Read-after-write gaps, conflicts and multi-step workflows - designing UX and undo paths for stale truth.

Before this: Idempotency and Retries

Objective

Design workflows that span multiple reliable steps: handle read-after-write gaps honestly, resolve concurrent-update conflicts deliberately, and build saga-style sequences where every step has a compensating partner.

Vocabulary

The words to use when directing your agent.

eventual consistency
A model where replicas converge over time - reads may briefly disagree with writes.
read-after-write gap
The window where a fresh write is not yet visible to subsequent reads.
conflict
Two writers updating the same data concurrently, requiring a resolution rule.
saga
A multi-step workflow where each step pairs with a compensation for rollback.
compensating action
The semantic undo of a completed step - refund the charge, release the booking.

Understand it

Living with the gap

Three practical tools. Route the author's next read to wherever their write landed (read-your-writes). Show pending/syncing states in UI instead of pretending instant truth. Timestamp displayed data so staleness is visible ('updated 30s ago'). None require exotic infrastructure - they require deciding, per feature, how much staleness users can feel and forgive.

Conflicts and compensations

Concurrent edits need a rule chosen in daylight: last-write-wins loses data quietly; version checks reject stale overwrites loudly; field-level merges preserve most intent. For multi-step flows, design the saga BEFORE shipping: enumerate steps, pair each with compensation, decide what happens when a compensation ITSELF fails - that escalation path goes to humans, visibly.

Core concepts

Stale reads are normal - design the UX for them

In distributed stores, the write you just made may not appear on the next read for a moment. Fighting this with denial produces bug reports; embracing it produces design: optimistic interfaces showing pending states, read-your-own-writes routing for the author, and labels admitting freshness honestly. The lie 'always instantly consistent everywhere' is unavailable at any price once you scale.

Every step needs its undo

Classic transactions span one database; real workflows span many services and cannot share one commit button. Sagas replace atomicity with choreography: each step commits locally and declares its compensating action - charge pairs with refund, reservation pairs with release. Failure mid-sequence triggers compensations in reverse, returning the world to honesty.

Examples

The disappearing comment

User posts a comment; the list refreshes from a lagging replica and shows nothing. Support ticket filed: 'your app ate my comment'. With a pending-state design, the same user sees their comment marked 'syncing' - truth arrives late but arrives visibly, trust intact.

Booking trip, hotel fails

Saga steps: book flight (committed), book hotel (fails). Compensation fires: cancel flight automatically, notify user honestly. No step pretends atomicity existed; the workflow's job is making the eventual state CORRECT even when intermediate states were messy.

Common mistakes

Common mistake

Assuming read-after-write everywhere because the demo worked on one node.

Fix

Identify which reads are author-sensitive and route/label them accordingly. Test on two replicas with real lag, not in imagination.

Common mistake

Multi-step workflows without designed compensations - 'we'll add undo later'.

Fix

No step ships without its named undo. If a step cannot be compensated, it must be idempotent-and-retryable or human-approved - decide explicitly.

Working with your AI agent

Force the saga table into existence before workflow code: step, action, compensation, failure-visibility, human-escalation path. Have your agent generate failure-injection tests for EVERY step boundary - between-step crashes are where orphaned states breed. UI copy matters too: draft the pending/stale messaging as part of the design, not as an afterthought.

Build it

The module’s real project. Your agent builds; you direct and inspect.

Project: Consistent Enough

Teach the Ledger honest distribution: a collaborator-invite saga with real compensations, conflict handling for concurrent edits, and UI states that admit staleness - proven by failure injection at every seam.

Requirements

  • Collaborator-invite saga: create invite, send notification (via your event wire), activate on acceptance
  • Every saga step has a working compensation; failing the notification step auto-cleans the invite
  • Compensation-failure path escalates visibly to a review queue, never silence
  • Concurrent-edit conflict handled: version check rejects stale writes with clear 409 messaging
  • UI admits staleness: pending states for async steps and freshness labels where reads may lag
  • Failure-injection suite exercises a fault between EVERY adjacent step pair

Context for your agent

Build collaborator invites on my Expense Ledger as an explicit saga: steps, actions, and named compensations agreed in a table BEFORE code. Notification rides my existing event wire. Inject failures at each step boundary - including compensation failure escalating to a review state. Add optimistic-concurrency version checks on expense edits returning helpful 409s. Design UI pending/stale states WITH me - wording included. Every claim gets a failure-injection test.

Verify it - twice

Check every item yourself first. Then hand your agent the prompt and compare verdicts.

Self-verified0/8
Functional
Architecture
Testing
Production

Verification prompt - paste into your AI agent

Probe my Consistent Enough build as a chaos engineer hunting orphaned state and lying interfaces. Execute: (1) kill the notification consumer mid-saga - did compensation fire and land somewhere visible?, (2) make a compensation ITSELF fail - trace the escalation path and confirm a human would find it Monday morning, (3) race two conflicting edits - is rejection honest, informative and loss-preserving for the newer intent?, (4) hunt for orphaned artifacts after each failed run - invites dangling, counters off-by-one, phantom rows. Grade every DoD item, then list any state my sagas can still strand with no path home.

Run this against “Eventual Consistency” after your own checklist pass. Where the agent and you disagree, investigate before moving on.

Review questions

Answer out loud. If you cannot, revisit before moving on.

  1. Which reads did YOU declare author-sensitive, and how are they routed or labeled?

  2. Recite your saga table: step, action, compensation - from memory.

  3. What happens when a compensation fails, specifically, in your build?

You can now

  • Design multi-step workflows with real compensations
  • Surface staleness honestly in interfaces and APIs
  • Resolve concurrent-edit conflicts by deliberate rule, not accident

Up next

Replication and Sharding