Eventual Consistency
Read-after-write gaps, conflicts and multi-step workflows - designing UX and undo paths for stale truth.
Before this: Idempotency and Retries
Objective
Design workflows that span multiple reliable steps: handle read-after-write gaps honestly, resolve concurrent-update conflicts deliberately, and build saga-style sequences where every step has a compensating partner.
Vocabulary
The words to use when directing your agent.
- eventual consistency
- A model where replicas converge over time - reads may briefly disagree with writes.
- read-after-write gap
- The window where a fresh write is not yet visible to subsequent reads.
- conflict
- Two writers updating the same data concurrently, requiring a resolution rule.
- saga
- A multi-step workflow where each step pairs with a compensation for rollback.
- compensating action
- The semantic undo of a completed step - refund the charge, release the booking.
Understand it
Living with the gap
Three practical tools. Route the author's next read to wherever their write landed (read-your-writes). Show pending/syncing states in UI instead of pretending instant truth. Timestamp displayed data so staleness is visible ('updated 30s ago'). None require exotic infrastructure - they require deciding, per feature, how much staleness users can feel and forgive.
Conflicts and compensations
Concurrent edits need a rule chosen in daylight: last-write-wins loses data quietly; version checks reject stale overwrites loudly; field-level merges preserve most intent. For multi-step flows, design the saga BEFORE shipping: enumerate steps, pair each with compensation, decide what happens when a compensation ITSELF fails - that escalation path goes to humans, visibly.
Core concepts
Stale reads are normal - design the UX for them
In distributed stores, the write you just made may not appear on the next read for a moment. Fighting this with denial produces bug reports; embracing it produces design: optimistic interfaces showing pending states, read-your-own-writes routing for the author, and labels admitting freshness honestly. The lie 'always instantly consistent everywhere' is unavailable at any price once you scale.
Every step needs its undo
Classic transactions span one database; real workflows span many services and cannot share one commit button. Sagas replace atomicity with choreography: each step commits locally and declares its compensating action - charge pairs with refund, reservation pairs with release. Failure mid-sequence triggers compensations in reverse, returning the world to honesty.
Examples
The disappearing comment
User posts a comment; the list refreshes from a lagging replica and shows nothing. Support ticket filed: 'your app ate my comment'. With a pending-state design, the same user sees their comment marked 'syncing' - truth arrives late but arrives visibly, trust intact.
Booking trip, hotel fails
Saga steps: book flight (committed), book hotel (fails). Compensation fires: cancel flight automatically, notify user honestly. No step pretends atomicity existed; the workflow's job is making the eventual state CORRECT even when intermediate states were messy.
Common mistakes
Common mistake
Assuming read-after-write everywhere because the demo worked on one node.
Fix
Identify which reads are author-sensitive and route/label them accordingly. Test on two replicas with real lag, not in imagination.
Common mistake
Multi-step workflows without designed compensations - 'we'll add undo later'.
Fix
No step ships without its named undo. If a step cannot be compensated, it must be idempotent-and-retryable or human-approved - decide explicitly.
Working with your AI agent
Force the saga table into existence before workflow code: step, action, compensation, failure-visibility, human-escalation path. Have your agent generate failure-injection tests for EVERY step boundary - between-step crashes are where orphaned states breed. UI copy matters too: draft the pending/stale messaging as part of the design, not as an afterthought.
Build it
The module’s real project. Your agent builds; you direct and inspect.
Project: Consistent Enough
Teach the Ledger honest distribution: a collaborator-invite saga with real compensations, conflict handling for concurrent edits, and UI states that admit staleness - proven by failure injection at every seam.
Requirements
- Collaborator-invite saga: create invite, send notification (via your event wire), activate on acceptance
- Every saga step has a working compensation; failing the notification step auto-cleans the invite
- Compensation-failure path escalates visibly to a review queue, never silence
- Concurrent-edit conflict handled: version check rejects stale writes with clear 409 messaging
- UI admits staleness: pending states for async steps and freshness labels where reads may lag
- Failure-injection suite exercises a fault between EVERY adjacent step pair
Context for your agent
Build collaborator invites on my Expense Ledger as an explicit saga: steps, actions, and named compensations agreed in a table BEFORE code. Notification rides my existing event wire. Inject failures at each step boundary - including compensation failure escalating to a review state. Add optimistic-concurrency version checks on expense edits returning helpful 409s. Design UI pending/stale states WITH me - wording included. Every claim gets a failure-injection test.
Verify it - twice
Check every item yourself first. Then hand your agent the prompt and compare verdicts.
Verification prompt - paste into your AI agent
Probe my Consistent Enough build as a chaos engineer hunting orphaned state and lying interfaces. Execute: (1) kill the notification consumer mid-saga - did compensation fire and land somewhere visible?, (2) make a compensation ITSELF fail - trace the escalation path and confirm a human would find it Monday morning, (3) race two conflicting edits - is rejection honest, informative and loss-preserving for the newer intent?, (4) hunt for orphaned artifacts after each failed run - invites dangling, counters off-by-one, phantom rows. Grade every DoD item, then list any state my sagas can still strand with no path home.
Run this against “Eventual Consistency” after your own checklist pass. Where the agent and you disagree, investigate before moving on.
Review questions
Answer out loud. If you cannot, revisit before moving on.
Which reads did YOU declare author-sensitive, and how are they routed or labeled?
Recite your saga table: step, action, compensation - from memory.
What happens when a compensation fails, specifically, in your build?
You can now
- Design multi-step workflows with real compensations
- Surface staleness honestly in interfaces and APIs
- Resolve concurrent-edit conflicts by deliberate rule, not accident
Up next
Replication and Sharding